# Electric corrections to $\pi$-$\pi$ scattering lenghts in the linear sigma
model

R. Cádiz  Facultad de Física, Pontificia Universidad Católica de Chile, Vicuña
Mackenna 4860, Santiago, Chile.  M. Loewe  Facultad de Ingeniería,
Arquitectura y Diseño, Universidad San Sebastián, Santiago, Chile.  Centre for
Theoretical and Mathematical Physics, and Department of Physics, University of
Cape Town, Rondebosch 7700, South Africa  R. Zamora  Instituto de Ciencias
Básicas, Universidad Diego Portales, Casilla 298-V, Santiago, Chile.
Facultad de Medicina Veterinaria, Universidad San Sebastián, Santiago, Chile.

###### Abstract

In this article we analyze the role of an external electric field, in the weak
field approximation, on $\pi$-$\pi$ scattering lengths. The discussion is
presented in the frame of the linear sigma model. To achieve this, we take
into account all one-loop corrections in the $s$, $t$, and $u$ channels
associated with the insertion of a Schwinger propagator for charged pions,
focusing on the region characterized by small values of the electric field.
Furthermore, one of the novelties of our work is the explicit calculation of
box diagrams, which were previously overlooked in discussions regarding
magnetic corrections. It turns out that the electric field corrections have an
opposite effect with respect to magnetic corrections calculated previously in
the literature.

##  I Introduction

During the last years, several research proposals have concentrated their
attention on the response of matter under extreme conditions due to external
agents. A clear example of this corresponds to relativistic heavy ion
collisions, like Au-Au, where extreme high temperatures are produced, allowing
for different phase transitions like deconfinement or chiral symmetry
restoration. Density effects might be also taken into account, producing an
interesting and rich phase diagram in the temperature-density plane. The
latter scenario can be found in compact objects like neutron stars. Also, a
quite interesting role is played by the extremely high magnetic field produced
during the very first stages of the collision between two heavy nuclei like
Au-Au. In the case of a relativistic collision between a heavy and a light
nuclei, for example Au-Cu, also an electric field appears, due to the
imbalance in the number of protons associated to each nucleus. Both fields,
the electric and the magnetic one, are produced in a relative perpendicular
configuration. The magnetic field points essentially perpendicular to the
collision plane, whereas the electric field points along the collision plane.
In the existing literature, there are several works related to the study of
different physical parameters in the presence of a magnetic and/or electric
field [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].

It is important to elucidate the relative effects associated to these external
agents on physical, in principle measurable, quantities. For example we know
that temperature and magnetic field conspire again each other in several
scenarios. Here we want to consider the effect of an external weak electric
field on $\pi$-$\pi$ scattering lengths, being these pions produced during the
collision. For this purpose we will work in the frame of the linear sigma
model, computing all relevant corrections to the scattering lengths. Fermion
contributions are neglected in the analysis since they are much more massive
than the other particles involved in the model: pions and the scalar sigma
field. It is important to avoid the strong field case since Schwinger
instabilities associated to pair productions of pions could appear. This
scenario goes beyond the present discussion.

Our discussion also considered the effects of box diagrams, previously not
taken into account [21, 22, 23] due the the relative high mass of the sigma
field. Although the influence of those diagrams, as expected, is in fact
small, they could play a relevant role when summing up, for example, ladder
diagrams with boxes, looking for reggeized amplitudes dependent on external
agents.

This analysis of $\pi$-$\pi$ scattering lengths corrections has been carried
out previously, using the same model, for the magnetic case [21, 22, 23]. As
we will see, the electric field corrections turn out to be opposite respect to
the equivalent magnetic corrections. This is an indication that it will be not
an easy task to isolate, in a clear way, the influence a certain specific
external agent on physical observables.

This article is organized as follows: In section II we present the linear
sigma model concentrating on the general structure of the $\pi$-$\pi$
scattering amplitudes an their projection into different isospin channels. In
section III we present the propagator for a boson immersed in a constant
external electric field showing also its weak field limit. We then go into the
different relevant diagrams and the techniques used for their evaluation.
Details are given in the appendices. Finally, we present our conclusions.

##  II Linear sigma model and $\pi$-$\pi$ Scattering

Gell-Mann and Lévy [24] proposed the linear sigma model (LSM) as an effective
framework to elucidate chiral symmetry breaking through both explicit and
spontaneous mechanisms. In the chiral broken phase the model is expressed as

$\displaystyle\mathcal{L}=\bar{\psi}\left[i\gamma^{\mu}\partial_{\mu}-m_{\psi}-g(\sigma+i\vec{\pi}\cdot\vec{\tau}\gamma_{5})\right]\psi$
$\displaystyle+\frac{1}{2}\left[(\partial\vec{\pi})^{2}+m_{\pi}^{2}\vec{\pi}^{2}\right]+\frac{1}{2}\left[(\partial\sigma)^{2}+m_{\sigma}^{2}\sigma^{2}\right]$
$\displaystyle-\lambda^{2}v\sigma(\sigma^{2}+\vec{\pi}^{2})-\frac{\lambda^{2}}{4}(\sigma^{2}+\vec{\pi}^{2})^{2}+(\varepsilon
c-vm_{\pi}^{2})\sigma.$ (1)

Pions are described by an isospin triplet,
$\vec{\pi}=(\pi_{1},\pi_{2},\pi_{3})$, $c\sigma$ is the term that breaks
explicitly the $SU(2)\times SU(2)$ chiral symmetry, being $\sigma$ a scalar
field, and $\varepsilon$ is a small dimensionless parameter. The model also
incorporates a doublet of Fermi fields, associated in the original version to
nucleon states, which in our context will be ignored since they are too heavy
as compared to the scalar sigma meson mass and to the relevant energy scale.
It is intriguing to note that the masses of all fields in the model are
determined by $v$. Indeed, the following relations can be proved to be valid:
$m_{\psi}=gv$, $m_{\pi}^{2}=\mu^{2}+\lambda^{2}v^{2}$ and
$m_{\sigma}^{2}=\mu^{2}+3\lambda^{2}v^{2}$. Perturbation theory at the tree
level allows us to identify the pion decay constants as $f_{\pi}=v$.

The LSM turns out to be a wonderful scenario for exploring effects of external
agents like temperature, magnetic field, electric field, and vorticity. These
effects have been studied in a series of articles by various authors, [25, 26,
27, 28]. In the present work we will explore, in the frame of the LSM model,
how an external electric field, generated in collisions between heavy and a
light nuclei, as for example Au-Cu collisions, will affect the $\pi$-$\pi$
scattering lengths. We will compare our results with previous analysis where
instead a magnetic field was considered.

The most general decomposition for the scattering amplitude for particles with
definite isospin quantum numbers is given by [29, 30]

$\displaystyle T_{\alpha\beta;\delta\gamma}$ $\displaystyle=$ $\displaystyle
A(s,t,u)\delta_{\alpha\beta}\delta_{\gamma\varepsilon}+A(t,s,u)\delta_{\alpha\varepsilon}\delta_{\beta\gamma}$
(2) $\displaystyle+A(u,t,s)\delta_{\alpha\gamma}\delta_{\beta\varepsilon},$

where $\alpha$, $\beta$, $\gamma$, $\delta$ represent isospin components.

Through the utilization of suitable projection operators

$\displaystyle P_{0}$
$\displaystyle=\dfrac{1}{3}\delta_{\alpha\beta}\delta_{\gamma\varepsilon},$
(3) $\displaystyle P_{1}$
$\displaystyle=-\dfrac{1}{2}\quantity(\delta_{\alpha\gamma}\delta_{\beta\varepsilon}-\delta_{\alpha\varepsilon}\delta_{\beta\gamma}),$
(4) $\displaystyle P_{2}$
$\displaystyle=\dfrac{1}{2}\quantity(\delta_{\alpha\gamma}\delta_{\beta\varepsilon}+\delta_{\alpha\varepsilon}\delta_{\beta\gamma}-\dfrac{2}{3}\delta_{\alpha\beta}\delta_{\gamma\varepsilon}),$
(5)

it is possible to find the following isospin dependent scattering amplitudes

$\displaystyle T^{0}$ $\displaystyle=3A(s,t,u)+A(t,s,u)+A(u,t,s),$ (6)
$\displaystyle T^{1}$ $\displaystyle=A(t,s,u)-A(u,t,s),$ (7) $\displaystyle
T^{2}$ $\displaystyle=A(t,s,u)+A(u,t,s),$ (8)

where $T^{I}$ denotes a scattering amplitude in a given isospin channel
$I=\\{0,1,2\\}$.

As is commonly understood [29], below the inelastic threshold any scattering
amplitude can be expanded in terms of partial amplitudes parameterized by
phase shifts for each angular momentum channel $\ell$. Hence, in the low-
energy region the isospin dependent scattering amplitude can be expanded in
partial wave components $T_{\ell}^{I}$. The real part of such an amplitude

$\real\left(T_{\ell}^{I}\right)=\left(\frac{p^{2}}{m_{\pi}^{2}}\right)^{\ell}\left(a_{\ell}^{I}+\frac{p^{2}}{m_{\pi}^{2}}b_{\ell}^{I}+\ldots\right),$
(9)

is normally expressed in terms of the scattering lengths $a_{\ell}^{I}$, and
the scattering slopes $b_{\ell}^{I}$, respectively. The scattering lengths
satisfy the hierarchy $|a_{0}^{I}|>|a_{1}^{I}|>|a_{2}^{I}|...$. Specifically,
in order to obtain the scattering lengths $a_{0}^{I}$, it is sufficient to
calculate the scattering amplitude $T^{I}$ in the static limit, i.e., when
$s\to 4m_{\pi}^{2}$, $t\to 0$ and $u\to 0$,

$a_{0}^{I}=\frac{1}{32\pi}T^{I}\left(s\to 4m_{\pi}^{2},t\to 0,u\to 0\right).$
(10)

The first measurement of $\pi$-$\pi$ scattering lengths was carried on by
Rosellet et al. [31]. More recently, these parameters have been measured using
pionium atoms in the DIRAC experiment [32], as well as through the decay of
heavy quarkonium states into $\pi$-$\pi$ final states, where the so called
cusp-effect was found [33].

##  III Scattering lengths at finite electric field

In previous works, we have computed the magnetic and thermal dependence of the
$\pi$-$\pi$ scattering lengths within the framework of the linear sigma model
[21, 22, 23]. In this instance, our aim to employ the same model exploring the
electric field dependence of the $\pi$-$\pi$ scattering lengths. For this
purpose, we will use the bosonic scalar propagator in the presence of an
electric field [34, 35], given by

$D(p)=\int_{0}^{\infty}{ds}\frac{e^{-s\left(\frac{\tanh(qiEs)}{qiEs}p_{\parallel}^{2}+p_{\perp}^{2}+m^{2}\right)}}{\cosh(qiEs)},$
(11)

where $q$ is the electric charge, $p_{\parallel}$ and $p_{\perp}$ refer to
$(p_{4},0,0,p_{3})$ and $(0,p_{1},p_{2},0)$, respectively. For simplicity, the
electric field points along the z-axis. Note that in the euclidean version
$p^{2}=p_{\parallel}^{2}+p_{\perp}^{2}$=$p_{4}^{2}+p_{3}^{2}+p_{1}^{2}+p_{2}^{2}$.
We are interested in the weak electric field region, since for a strong
electric field the Schwinger effect might appear, i.e., the generation of
electron-positron pairs. Therefore we will proceed to expand the previous
expression up to order $\mathcal{O}(E^{2})$, to obtain

$\displaystyle D(p)\approx\frac{1}{p^{2}+m^{2}}$
$\displaystyle-(qE)^{2}\left(-\frac{1}{(p^{2}+m^{2})^{3}}+\frac{2p_{\parallel}^{2}}{(p^{2}+m^{2})^{4}}\right).$
(12)

However, using the relation $p^{2}=p_{\parallel}^{2}+p_{\perp}^{2}$, Eq. (12)
can be written as

$\displaystyle D(p)$
$\displaystyle\approx\dfrac{1}{p^{2}+m^{2}}+\dfrac{q^{2}E^{2}\quantity[2\quantity(p_{\perp}^{2}+m^{2})-\quantity(p^{2}+m^{2})]}{\quantity(p^{2}+m^{2})^{4}}.$
(13)

The above expression, being more symmetric, is useful to carry on the
integrals that will appear when computing all relevant loop corrections. Also,
to distinguish between the free and charged propagators, we will define the
free scalar propagator as

$S(p)=\dfrac{1}{p^{2}+m^{2}}.$ (14)

###  III.1 Loop Integrals Classification

For the analysis, we need to compute 21 Feynman diagrams. The diagrams that
contribute to the $s$ channel are shown in figure 1, and those diagrams
relevant for the $t$ channel can be seen in figure 2. The $u$ channel diagrams
are analogous to the $t$ channel. The only difference corresponds to a
permutation of isospin indexes of the external legs.

(a)  (b)

(c)

(d)  (e)

(f)

(g)  (h)

(i)

(j)

Figure 1: $s$ channel diagrams.

(a)  (b)

(c)  (d)

(e)  (f)

(g)  (h)

(i)  (j)

(k)

Figure 2: $t$ channel diagrams.

In previous works [21, 22, 23], and because the sigma boson mass is much
bigger than the pion mass, considering also the static limit approximation,
the diagrams that contained sigma bosons were pinched, i.e., the sigma
propagator was contracted to a point. The following limit was used

$\dfrac{1}{p^{2}+m_{\sigma}^{2}}\longrightarrow\dfrac{1}{m_{\sigma}^{2}}.$
(15)

Pinching the sigma propagators simplify several diagrams, and all the
calculations can be reduced to computing only five integrals.

In the present analysis we present a detailed discussion of all diagrams,
including full sigma propagators. This implies the calculation of several box
diagrams which have not been considered in our previous articles.

Before going into the details of the calculation, it is useful to classify all
integrals into different types of integrals, which will be called $I_{i}$,
where $i=1,\dots,9$ represents the $i-$type integral, except for the case of
$I_{B1}$ and $I_{B2}$, that represents the integrals associated to the box
diagrams 1 and 2, respectively. These integrals can be found in appendix A,
where only terms up to the order $\mathcal{O}(E^{2})$ were considered.

###  III.2 Mathematical Methods

For the purpose of computing the different loop corrections, essentially two
methods were used. For some integrals the standard dimensional regularization
was employed.

For the remaining integrals, which are the majority of cases, four dimensional
hyper-spherical coordinates were used. In particular, the calculations of the
first box diagram can be found in appendix B. Based on [36], the set of
coordinates used were

$\begin{cases}x_{0}=r\cos\theta_{1}\\\\[7.0pt]
x_{1}=r\sin\theta_{1}\cos\theta_{2}\\\\[7.0pt]
x_{2}=r\sin\varphi\sin\theta_{1}\sin\theta_{2}\\\\[7.0pt]
x_{3}=r\cos\varphi\sin\theta_{1}\sin\theta_{2},\end{cases}$ (16)

where $0\leq\varphi\leq 2\pi$, $0\leq\theta_{i}\leq\pi$, $0\leq r<\infty$, and
the jacobian of the transformation given by
$J=r^{3}\sin^{2}\theta_{1}\sin\theta_{2}$.

Along with the change of coordinates, it is important to note that a frame of
reference can be set, without loss of generality, where the four momenta $p$
takes the form $p_{\mu}=\quantity(m_{\pi},\mathbf{0})$, selecting a privileged
direction according to

$k\cdot p=m_{\pi}r\cos\theta_{1}.$ (17)

###  III.3 Isospin Projections

Because of the associated Feynman rules, all the integrals emerging from the
diagrams have a determined isospin structure. These structures are simplified
when isospin projection operators acting on the different integrals are used.

Using the projection operators (3), (4), and (5), all projections can be
easily obtained. The numerical factors associated to each one of them can be
seen in the following table 1.

Isospin Structure | $P_{0}$ | $P_{1}$ | $P_{2}$
---|---|---|---
$7\delta_{\alpha\gamma}\delta_{\beta\varepsilon}+2\delta_{\alpha\beta}\delta_{\gamma\varepsilon}+2\delta_{\alpha\varepsilon}\delta_{\beta\gamma}$ | $15$ | $-15$ | $45$
$7\delta_{\alpha\beta}\delta_{\gamma\varepsilon}+2\delta_{\alpha\gamma}\delta_{\beta\varepsilon}+2\delta_{\alpha\varepsilon}\delta_{\beta\gamma}$ | $25$ | $0$ | $20$
$\delta_{\alpha\beta}\delta_{\gamma\varepsilon}+\delta_{\alpha\gamma}\delta_{\beta\varepsilon}+\delta_{\alpha\varepsilon}\delta_{\beta\gamma}$ | $5$ | $0$ | $10$
$\delta_{\alpha\beta}\delta_{\gamma\varepsilon}$ | $3$ | $0$ | $0$
$\delta_{\alpha\gamma}\delta_{\beta\varepsilon}$ | $1$ | $-3$ | $5$
$7\delta_{\alpha\varepsilon}\delta_{\beta\gamma}+2\delta_{\alpha\beta}\delta_{\gamma\varepsilon}+2\delta_{\alpha\gamma}\delta_{\beta\varepsilon}$ | $15$ | $15$ | $45$
$\delta_{\alpha\varepsilon}\delta_{\beta\gamma}$ | $1$ | $3$ | $5$
$\delta_{\alpha\beta}\delta_{\gamma\varepsilon}+\delta_{\alpha\varepsilon}\delta_{\beta\gamma}+\delta_{\alpha\gamma}\delta_{\beta\varepsilon}$ | $5$ | $0$ | $10$
Table 1: Table of factors obtained from the different isospin structure when
acting with the isospin projectors.

After this computation, it is interesting to note that both the $t$ and $u$
channel give the same results, except for an opposite sign. Therefore, our
analysis leads to the following scattering amplitudes in the three isospin
channels.

$\displaystyle T^{0}$ $\displaystyle=3A(s,t,u)+2A(t,s,u),$ (18) $\displaystyle
T^{1}$ $\displaystyle=0,$ (19) $\displaystyle T^{2}$
$\displaystyle=2A(t,s,u).$ (20)

Using expressions 18, 19 and 20, along with the projections reported in table
1, it is found that the scattering amplitudes get the following form.

$\displaystyle T^{0}$
$\displaystyle=32\lambda^{8}v^{4}\quantity(\dfrac{9}{2}I_{B1}+I_{B2}+I_{9})+I_{3}\left(\frac{144\lambda^{8}v^{4}}{4m_{\pi}^{2}+m_{\sigma}^{2}}+120\lambda^{6}v^{2}\right)+I_{4}\left(\frac{32\lambda^{8}v^{4}}{m_{\sigma}^{2}}+80\lambda^{6}v^{2}\right)+I_{6}\left(\frac{432\lambda^{8}v^{4}}{4m_{\pi}^{2}+m_{\sigma}^{2}}+72\lambda^{6}v^{2}\right)$
$\displaystyle\hskip
28.45274pt+I_{5}\left(\frac{96\lambda^{8}v^{4}}{m_{\sigma}^{2}}+16\lambda^{6}v^{2}\right)+I_{7}\left(\frac{216\lambda^{6}v^{3}}{\left(4m_{\pi}^{2}+m_{\sigma}^{2}\right){}^{2}}+\frac{48\lambda^{6}v^{3}}{M^{4}}\right)+I_{8}\left(\frac{72\lambda^{6}v^{2}}{4m_{\pi}^{2}+m_{\sigma}^{2}}+\frac{16\lambda^{6}v^{2}}{m_{\sigma}^{2}}\right)$
$\displaystyle\hskip
28.45274pt+I_{2}\left(300\lambda^{4}+\frac{432\lambda^{8}v^{4}}{\left(4m_{\pi}^{2}+m_{\sigma}^{2}\right){}^{2}}+\frac{360\lambda^{6}v^{3}}{4m_{\pi}^{2}+m_{\sigma}^{2}}\right)+I_{1}\left(120\lambda^{4}+\frac{96\lambda^{8}v^{4}}{m_{\sigma}^{4}}+\frac{80\lambda^{6}v^{3}}{m_{\sigma}^{2}}\right),$
(21) $\displaystyle T^{2}$
$\displaystyle=160\lambda^{8}v^{4}\quantity(I_{B2}+I_{9})+I_{4}\left(\frac{160\lambda^{8}v^{4}}{m_{\sigma}^{2}}+160\lambda^{6}v^{2}\right)+I_{5}\left(\frac{480\lambda^{8}v^{4}}{m_{\sigma}^{2}}+80\lambda^{6}v^{2}\right)$
$\displaystyle\hskip
28.45274pt+\frac{240\lambda^{6}v^{3}}{m_{\sigma}^{4}}I_{7}+\frac{80\lambda^{6}v^{2}}{m_{\sigma}^{2}}I_{8}+I_{1}\left(360\lambda^{4}+\frac{480\lambda^{8}v^{4}}{m_{\sigma}^{4}}+\frac{400\lambda^{6}v^{3}}{m_{\sigma}^{2}}\right).$
(22)

In order to obtain the scattering lengths $a_{0}^{I}$, using Eq. (10), we get

$\displaystyle a_{0}^{0}(E)$ $\displaystyle=$ $\displaystyle
a_{0}^{0}(\text{exp})+\frac{1}{32\pi}T^{0},$ $\displaystyle a_{0}^{2}(E)$
$\displaystyle=$ $\displaystyle a_{0}^{2}(\text{exp})+\frac{1}{32\pi}T^{2},$
(23)

where $T^{0}$ and $T^{2}$ are given by Eq. (21) and Eq. (22), respectively.
The experimental values at tree level are determined by [37]
$a_{0}^{0}(\text{exp})=0.217$ and $a_{0}^{2}(\text{exp})=-0.041$.

In order to discuss the behavior of scattering lengths in the presence of an
electric field, we will employ Eq. (23) normalizing to experimental values. We
will use the following parameters $m_{\pi}=$140\text{\,}\mathrm{MeV}$$,
$m_{\sigma}=$550\text{\,}\mathrm{MeV}$$, $v=$89\text{\,}\mathrm{MeV}$$, and
$\lambda^{2}=4.26$, obtaining the following plot 3.

Figure 3: Behaviour of the normalized scattering lengths as a function of
$qE/m_{\pi}^{2}$. Red lines represent $a_{0}^{0}(E)/a_{0}^{0}(\text{exp})$,
while blue lines represent $a_{0}^{2}(E)/a_{0}^{2}(\text{exp})$.

##  IV Conclusions

We have presented an analytic calculation of $\pi$-$\pi$ scattering lengths
within the linear sigma model at the one-loop level in the isospin channels
$I=\\{0,2\\}$, as a function of the external electric field intensity. We
provide a plot illustrating the calculation of scattering lengths $a_{0}^{0}$
and $a_{0}^{2}$ as a function of the electric field. It is evident that
$a_{0}^{0}$ increases with the electric field, while $a_{0}^{2}$ decreases.
This behavior is interesting, as it contrasts sharply with the effect of an
external magnetic field. Such opposition between electric and magnetic fields
is also observed in the calculation of renormalons [34]. A significant novelty
of this calculation is our inclusion of box diagrams, which have been fully
computed, unlike in previous works [21, 22, 23].

###### Acknowledgements.

R. C. acknowledges support from ANID/CONICYT FONDECYT Regular (Chile) under
Grant No. 1220035. M. L. acknowledges support from ANID/CONICYT FONDECYT
Regular (Chile) under Grant No. 1241436 and 1220035. R. Z. acknowledges
support from ANID/CONICYT FONDECYT Regular (Chile) under Grant No. 1241436 and
1220035.

##  Appendix A Types of Integrals

As stated in section III.1, by analyzing all the integrals found from the
Feynman diagrams, we can classify them into several types, as shown below.

  1. $\displaystyle I_{1}\equiv\int\dfrac{\differential[4]{k}}{(2\pi)^{4}}D(k)^{2}$,

  2. $\displaystyle I_{2}\equiv\int\dfrac{\differential[4]{k}}{(2\pi)^{4}}D(k)D(k-2p)$,

  3. $\displaystyle I_{3}\equiv\int\dfrac{\differential[4]{k}}{(2\pi)^{4}}S(k)D(p+k)D(p-k)$,

  4. $\displaystyle I_{4}\equiv\int\dfrac{\differential[4]{k}}{(2\pi)^{4}}S(k)D(p-k)^{2}$,

  5. $\displaystyle I_{5}\equiv\int\dfrac{\differential[4]{k}}{(2\pi)^{4}}S(p-k)^{2}D(k)$,

  6. $\displaystyle I_{6}\equiv\int\dfrac{\differential[4]{k}}{(2\pi)^{4}}S(p+k)S(p-k)D(k)$,

  7. $\displaystyle I_{7}\equiv\int\dfrac{\differential[4]{k}}{(2\pi)^{4}}D(k)$,

  8. $\displaystyle I_{8}\equiv\int\dfrac{\differential[4]{k}}{(2\pi)^{4}}S(p\mp k)D(k)$,

  9. $\displaystyle I_{9}\equiv\int\dfrac{\differential[4]{k}}{(2\pi)^{4}}S(p-k)^{2}D(k)^{2}$,

  10. $\displaystyle I_{B1}\equiv\int\dfrac{\differential[4]{k}}{(2\pi)^{4}}S(p+k)S(p-k)D(k)^{2}$,

  11. $\displaystyle I_{B2}\equiv\int\dfrac{\differential[4]{k}}{(2\pi)^{4}}S(p-k)^{2}D(k)D(2p-k)$,

with $D(p)$ and $S(p)$ is defined in Eq. (12) and Eq. (14) respectively.

Also, in $I_{8}$, the change of sign does not produce a new class of
integrals.

##  Appendix B Box Diagram Calculation

As mentioned in section III.2, one of the innovations of this work is the
calculation of the full box diagrams. Let’s begin by analyzing the first one
of them. In the $s$ channel, we have

$\alpha$$\gamma$$\varepsilon$$\beta$$p$$p+k$$p$$k$$k$$p$$p-k$$p$ Figure 4: Box
diagram 1 associated to the $s$ channel.

By means of an adequate isospin parametrization, together with the Feynman
rules, the associated integral $I_{B1}$ gets the following form

$I_{B1}=16\lambda^{8}v^{4}\delta_{\alpha\beta}\delta_{\gamma\varepsilon}\int\dfrac{\differential[4]{k}}{(2\pi)^{4}}\>D(k)^{2}S(k+p)S(k-p).$
(24)

After expanding the propagators up to $\mathcal{O}(E^{2})$, we get

$\displaystyle
D(k)^{2}S(p+k)S(p-k)=\dfrac{1}{\quantity(k^{2}+m_{\pi}^{2})^{2}\quantity(\quantity(p+k)^{2}+m_{\sigma}^{2})\quantity(\quantity(p-k)^{2}+m_{\sigma}^{2})}$
$\displaystyle\hskip
14.22636pt+2q^{2}E^{2}\Bigg{[}\dfrac{2\quantity(k_{\perp}^{2}+m_{\pi}^{2})}{\quantity(k^{2}+m_{\pi}^{2})^{5}\quantity(\quantity(p+k)^{2}+m_{\sigma}^{2})\quantity(\quantity(p-k)^{2}+m_{\sigma}^{2})}-\dfrac{1}{\quantity(k^{2}+m_{\pi}^{2})^{4}\quantity(\quantity(p+k)^{2}+m_{\sigma}^{2})\quantity(\quantity(p-k)^{2}+m_{\sigma}^{2})}\Bigg{]}+O(E^{3}).$
(25)

To continue our calculation, from Eq. (25), we have to deal with three
different integrals. These calculations were performed using the hyper-
spherical coordinates introduced in section III.2. We obtain

$\displaystyle\mathcal{I}_{1}$
$\displaystyle=\int\dfrac{1}{\quantity(r^{2}+m_{\pi}^{2})^{2}\quantity(r^{2}+2m_{\pi}r\cos\theta_{1}+m_{\pi}^{2}+m_{\sigma}^{2})\quantity(r^{2}-2m_{\pi}r\cos\theta_{1}+m_{\pi}^{2}+m_{\sigma}^{2})}\cdot
r^{3}\differential{r}\differential{\Omega_{4}},$ (26)
$\displaystyle\mathcal{I}_{2}$
$\displaystyle=\int\dfrac{\quantity(r\sin\varphi\sin\theta_{1}\sin\theta_{2})^{2}+\quantity(r\sin\theta_{1}\cos\theta_{2})^{2}+m^{2}}{\quantity(r^{2}+m_{\pi}^{2})^{5}\quantity(r^{2}+2m_{\pi}r\cos\theta_{1}+m_{\pi}^{2}+m_{\sigma}^{2})\quantity(r^{2}-2m_{\pi}r\cos\theta_{1}+m_{\pi}^{2}+m_{\sigma}^{2})}\cdot
r^{3}\differential{r}\differential{\Omega_{4}},$ (27)
$\displaystyle\mathcal{I}_{3}$
$\displaystyle=\int\dfrac{1}{\quantity(r^{2}+m_{\pi}^{2})^{4}\quantity(r^{2}+2m_{\pi}r\cos\theta_{1}+m_{\pi}^{2}+m_{\sigma}^{2})\quantity(r^{2}-2m_{\pi}r\cos\theta_{1}+m_{\pi}^{2}+m_{\sigma}^{2})}\cdot
r^{3}\differential{r}\differential{\Omega_{4}},$ (28)

where
$\differential{\Omega_{4}}=\sin^{2}\theta_{1}\sin\theta_{2}\differential{\theta_{1}}\differential{\theta_{2}}$
represents the angular measure. It is important to note that the angular
integration can be carried out without major problems. After performing all
the integrals, we find the following results.

$\displaystyle\mathcal{I}_{1}$
$\displaystyle=\dfrac{2m_{\pi}\left(2m_{\pi}^{2}+m_{\sigma}^{2}\right)\left(\tanh^{-1}\frac{m_{\sigma}^{2}}{\sqrt{4m_{\pi}^{4}+m_{\sigma}^{4}}}+\ln\frac{\sqrt{4m_{\pi}^{4}+m_{\sigma}^{4}}+2m_{\pi}^{2}-m_{\sigma}^{2}}{\sqrt{4m_{\pi}^{4}+m_{\sigma}^{4}}-2m_{\pi}^{2}+m_{\sigma}^{2}}\right)-\sqrt{4m_{\pi}^{6}+4m_{\pi}^{4}m_{\sigma}^{2}+m_{\pi}^{2}m_{\sigma}^{4}+m_{\sigma}^{6}}\ln\left(\frac{\sqrt{m_{\pi}^{2}+m_{\sigma}^{2}}+m}{\sqrt{m_{\pi}^{2}+m_{\sigma}^{2}}-m_{\pi}}\right)}{16\pi^{2}m_{\pi}m_{\sigma}^{4}\sqrt{4m_{\pi}^{4}+m_{\sigma}^{4}}},$
(29) $\displaystyle\mathcal{I}_{2}$
$\displaystyle=\dfrac{1}{2304\pi^{2}m_{\pi}^{12}m_{\sigma}^{10}\left(m_{\sigma}^{4}+4m_{\pi}^{4}\right){}^{7/2}}\Bigg{[}m_{\sigma}^{10}\left(-8m_{\pi}^{2}m_{\sigma}^{2}-3m_{\sigma}^{4}+18m_{\pi}^{4}\right)\left(m_{\sigma}^{4}+4m_{\pi}^{4}\right){}^{7/2}$
$\displaystyle\hskip
5.69046pt+m_{\sigma}^{4}\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}\bigg{(}-1536m_{\pi}^{20}m_{\sigma}^{2}+480m_{\pi}^{18}m_{\sigma}^{4}-1536m_{\pi}^{16}m_{\sigma}^{6}-160m_{\pi}^{14}m_{\sigma}^{8}-420m_{\pi}^{12}m_{\sigma}^{10}$
$\displaystyle\hskip
142.26378pt+288m_{\pi}^{10}m_{\sigma}^{12}-54m_{\pi}^{8}m_{\sigma}^{14}+96m_{\pi}^{6}m_{\sigma}^{16}+18m_{\pi}^{4}m_{\sigma}^{18}+8m_{\pi}^{2}m_{\sigma}^{20}+3m_{\sigma}^{22}+768m_{\pi}^{22}\bigg{)}$
$\displaystyle\hskip
5.69046pt+48m_{\pi}^{11}\sqrt{m_{\sigma}^{2}+m_{\pi}^{2}}\left(m_{\pi}^{2}-2m_{\sigma}^{2}\right)\left(m_{\sigma}^{4}+4m_{\pi}^{4}\right){}^{7/2}\ln\left(\frac{2m_{\pi}\left(\sqrt{m_{\sigma}^{2}+m_{\pi}^{2}}+m_{\pi}\right)}{m_{\sigma}^{2}}+1\right)$
$\displaystyle\hskip
5.69046pt-24m_{\pi}^{12}\left(-384m_{\pi}^{14}m_{\sigma}^{2}-64m_{\pi}^{12}m_{\sigma}^{4}-256m_{\pi}^{10}m_{\sigma}^{6}-224m_{\pi}^{8}m_{\sigma}^{8}-8m_{\pi}^{6}m_{\sigma}^{10}-114m_{\pi}^{4}m_{\sigma}^{12}+8m_{\pi}^{2}m_{\sigma}^{14}-9m_{\sigma}^{16}+256m_{\pi}^{16}\right)$
$\displaystyle\hskip
199.16928pt\cdot\left(2\tanh^{-1}\frac{m_{\sigma}^{2}}{\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}}-\ln\frac{m_{\sigma}^{2}+\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}-2m_{\pi}^{2}}{-m_{\sigma}^{2}+\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}+2m_{\pi}^{2}}\right)\Bigg{]},$
(30) $\displaystyle\mathcal{I}_{3}$
$\displaystyle=\dfrac{1}{96\pi^{2}m_{\pi}^{4}m_{\sigma}^{8}\left(m_{\sigma}^{4}+4m_{\pi}^{4}\right){}^{3}}\Bigg{[}m_{\sigma}^{4}\left(-32m_{\pi}^{8}m_{\sigma}^{4}-24m_{\pi}^{6}m_{\sigma}^{6}+2m_{\pi}^{4}m_{\sigma}^{8}-6m_{\pi}^{2}m_{\sigma}^{10}+m_{\sigma}^{12}-96m_{\pi}^{12}\right)$
$\displaystyle\hskip
5.69046pt-6m_{\pi}^{3}\sqrt{m_{\sigma}^{2}+m_{\pi}^{2}}\left(m_{\sigma}^{4}+4m_{\pi}^{4}\right){}^{3}\ln\left(\frac{2m_{\pi}\left(\sqrt{m_{\sigma}^{2}+m_{\pi}^{2}}+m_{\pi}\right)}{m_{\sigma}^{2}}+1\right)$
$\displaystyle\hskip
5.69046pt+12m_{\pi}^{4}\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}\left(16m_{\pi}^{8}m_{\sigma}^{2}+16m_{\pi}^{6}m_{\sigma}^{4}+12m_{\pi}^{4}m_{\sigma}^{6}+3m_{\sigma}^{10}+32m_{\pi}^{10}\right)\left(\tanh^{-1}\frac{m_{\sigma}^{2}}{\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}}+\tanh^{-1}\frac{2m_{\pi}^{2}-m_{\sigma}^{2}}{\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}}\right)\Bigg{]}.$
(31)

With expressions (29), (30) and (31), these can be combined and simplified to
obtain an analytical result for the total loop. In fact, from the expansion of
the propagators, it is found that we can write

$I_{B1}=-16\lambda^{8}v^{4}\delta_{\alpha\beta}\delta_{\gamma\varepsilon}\quantity[\mathcal{I}_{1}+2q^{2}E^{2}\quantity(2\mathcal{I}_{2}-\mathcal{I}_{3})]$
(32)

Therefore, and because we are interested in the electric corrections, the
vacuum term is not considered. From this, after replacing (30) and (31) into
(32), the terms proportional to $\quantity(qE)^{2}$ are given by the following
expression.

$\displaystyle I_{B1}$
$\displaystyle=-\dfrac{\lambda^{8}v^{4}\delta_{\alpha\beta}\delta_{\gamma\varepsilon}\quantity(qE)^{2}}{36\pi^{2}m_{\pi}^{12}m_{\sigma}^{10}\left(m_{\sigma}^{4}+4m_{\pi}^{4}\right){}^{7/2}}\Bigg{[}12m_{\pi}^{8}m_{\sigma}^{6}\left(2m_{\pi}^{4}m_{\sigma}^{4}+6m_{\pi}^{2}m_{\sigma}^{6}-m_{\sigma}^{8}+24m_{\pi}^{8}\right)\left(m_{\sigma}^{4}+4m_{\pi}^{4}\right){}^{3/2}$
$\displaystyle\hskip
5.69046pt+m_{\sigma}^{10}\left(-8m_{\pi}^{2}m_{\sigma}^{2}-3m_{\sigma}^{4}+18m_{\pi}^{4}\right)\left(m_{\sigma}^{4}+4m_{\pi}^{4}\right){}^{7/2}$
$\displaystyle\hskip
5.69046pt+\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}m_{\sigma}^{4}\bigg{(}-1536m_{\pi}^{20}m_{\sigma}^{2}+480m_{\pi}^{18}m_{\sigma}^{4}-1536m_{\pi}^{16}m_{\sigma}^{6}-160m_{\pi}^{14}m_{\sigma}^{8}-420m_{\pi}^{12}m_{\sigma}^{10}+288m_{\pi}^{10}m_{\sigma}^{12}$
$\displaystyle\hskip
227.62204pt-54m_{\pi}^{8}m_{\sigma}^{14}+96m_{\pi}^{6}m_{\sigma}^{16}+18m_{\pi}^{4}m_{\sigma}^{18}+8m_{\pi}^{2}m_{\sigma}^{20}+3m_{\sigma}^{22}+768m_{\pi}^{22}\bigg{)}$
$\displaystyle\hskip
5.69046pt+24m_{\pi}^{11}\sqrt{m_{\sigma}^{2}+m_{\pi}^{2}}\left(2m_{\pi}^{2}-m_{\sigma}^{2}\right)\left(m_{\sigma}^{4}+4m_{\pi}^{4}\right){}^{7/2}\ln\left(\frac{2m_{\pi}\left(\sqrt{m_{\sigma}^{2}+m_{\pi}^{2}}+m_{\pi}\right)}{m_{\sigma}^{2}}+1\right)$
$\displaystyle\hskip
5.69046pt-144m_{\pi}^{12}m_{\sigma}^{2}\left(16m_{\pi}^{8}m_{\sigma}^{2}+16m_{\pi}^{6}m_{\sigma}^{4}+12m_{\pi}^{4}m_{\sigma}^{6}+3m_{\sigma}^{10}+32m_{\pi}^{10}\right)\left(m_{\sigma}^{4}+4m_{\pi}^{4}\right)$
$\displaystyle\hskip
170.71652pt\cdot\left(\tanh^{-1}\frac{m_{\sigma}^{2}}{\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}}+\tanh^{-1}\frac{2m_{\pi}^{2}-m_{\sigma}^{2}}{\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}}\right)$
$\displaystyle\hskip
5.69046pt+24m_{\pi}^{12}\left(-384m_{\pi}^{14}m_{\sigma}^{2}-64m_{\pi}^{12}m_{\sigma}^{4}-256m_{\pi}^{10}m_{\sigma}^{6}-224m_{\pi}^{8}m_{\sigma}^{8}-8m_{\pi}^{6}m_{\sigma}^{10}-114m_{\pi}^{4}m_{\sigma}^{12}+8m_{\pi}^{2}m_{\sigma}^{14}-9m_{\sigma}^{16}+256m_{\pi}^{16}\right)$
$\displaystyle\hskip
170.71652pt\cdot\left(\ln\frac{m_{\sigma}^{2}+\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}-2m_{\pi}^{2}}{-m_{\sigma}^{2}+\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}+2m_{\pi}^{2}}-2\tanh^{-1}\frac{m_{\sigma}^{2}}{\sqrt{m_{\sigma}^{4}+4m_{\pi}^{4}}}\right)\Bigg{]}$
(33)

With this correction, the second box diagram can be computed in a completely
analogous way. However, the calculation for $I_{B2}$ is much longer, because
in that case, five integrals must be solved to get the needed analytical
result.

## References

  * Gusynin  _et al._ [1996] V. P. Gusynin, V. A. Miransky, and I. A. Shovkovy, Nucl. Phys. B 462, 249 (1996), arXiv:hep-ph/9509320 .
  * Miransky and Shovkovy [2002] V. A. Miransky and I. A. Shovkovy, Phys. Rev. D 66, 045006 (2002), arXiv:hep-ph/0205348 .
  * Shovkovy [2013] I. A. Shovkovy, Lect. Notes Phys. 871, 13 (2013), arXiv:1207.5081 [hep-ph] .
  * Bali  _et al._ [2012] G. S. Bali, F. Bruckmann, G. Endrodi, Z. Fodor, S. D. Katz, and A. Schafer, Phys. Rev. D 86, 071502 (2012), arXiv:1206.4205 [hep-lat] .
  * Bali  _et al._ [2014] G. S. Bali, F. Bruckmann, G. Endrödi, S. D. Katz, and A. Schäfer, JHEP 08, 177 (2014), arXiv:1406.0269 [hep-lat] .
  * Kogut and Sinclair [2002] J. B. Kogut and D. K. Sinclair, Phys. Rev. D 66, 034505 (2002), arXiv:hep-lat/0202028 .
  * Bazavov  _et al._ [2017] A. Bazavov _et al._ , Phys. Rev. D 95, 054504 (2017), arXiv:1701.04325 [hep-lat] .
  * Johnson and Kundu [2008] C. V. Johnson and A. Kundu, JHEP 12, 053 (2008), arXiv:0803.0038 [hep-th] .
  * Frasca and Ruggieri [2011] M. Frasca and M. Ruggieri, Phys. Rev. D 83, 094024 (2011), arXiv:1103.1194 [hep-ph] .
  * Andersen [2021] J. O. Andersen, Eur. Phys. J. A 57, 189 (2021), arXiv:2102.13165 [hep-ph] .
  * Bandyopadhyay and Farias [2021] A. Bandyopadhyay and R. L. S. Farias, Eur. Phys. J. ST 230, 719 (2021), arXiv:2003.11054 [hep-ph] .
  * Gaspar  _et al._ [2023] I. I. Gaspar, L. A. Hernández, and R. Zamora, Phys. Rev. D 108, 094020 (2023), arXiv:2305.00101 [hep-ph] .
  * Zayakin [2008] A. V. Zayakin, JHEP 07, 116 (2008), arXiv:0807.2917 [hep-th] .
  * Filev  _et al._ [2009] V. G. Filev, C. V. Johnson, and J. P. Shock, JHEP 08, 013 (2009), arXiv:0903.5345 [hep-th] .
  * Ballon-Bayona  _et al._ [2020] A. Ballon-Bayona, J. P. Shock, and D. Zoakos, JHEP 10, 193 (2020), arXiv:2005.00500 [hep-th] .
  * Ayala  _et al._ [2023] A. Ayala, R. L. S. Farias, L. A. Hernández, A. J. Mizher, J. Rendón, C. Villavicencio, and R. Zamora, (2023), arXiv:2311.13068 [hep-ph] .
  * Dominguez  _et al._ [2023] C. A. Dominguez, M. Loewe, C. Villavicencio, and R. Zamora, Phys. Rev. D 108, 074024 (2023), arXiv:2308.05663 [hep-ph] .
  * Castaño Yepes  _et al._ [2023] J. D. Castaño Yepes, M. Loewe, E. Muñoz, J. C. Rojas, and R. Zamora, Phys. Rev. D 107, 096014 (2023), arXiv:2211.16985 [hep-th] .
  * Loewe  _et al._ [2023] M. Loewe, D. Valenzuela, and R. Zamora, Eur. Phys. J. A 59, 184 (2023), arXiv:2207.12387 [hep-ph] .
  * Fernández  _et al._ [2024] G. Fernández, L. A. Hernández, and R. Zamora, (2024), arXiv:2403.14478 [hep-ph] .
  * Loewe  _et al._ [2019a] M. Loewe, E. Muñoz, and R. Zamora, Phys. Rev. D 100, 116006 (2019a), arXiv:1905.03783 [hep-ph] .
  * Loewe  _et al._ [2019b] M. Loewe, L. Monje, E. Muñoz, A. Raya, and R. Zamora, Phys. Rev. D 99, 056002 (2019b), arXiv:1901.03256 [hep-ph] .
  * Loewe  _et al._ [2018] M. Loewe, L. Monje, and R. Zamora, Phys. Rev. D 97, 056023 (2018), arXiv:1712.10047 [hep-ph] .
  * Gell-Mann and Levy [1960] M. Gell-Mann and M. Levy, Nuovo Cim. 16, 705 (1960).
  * Schaefer and Wagner [2009] B.-J. Schaefer and M. Wagner, Phys. Rev. D 79, 014018 (2009).
  * Kovács and Szép [2007] P. Kovács and Z. Szép, Phys. Rev. D 75, 025015 (2007).
  * Kovács and Szép [2008] P. Kovács and Z. Szép, Phys. Rev. D 77, 065016 (2008).
  * Kovács  _et al._ [2016] P. Kovács, Z. Szép, and G. Wolf, Phys. Rev. D 93, 114014 (2016).
  * Collins [2023] P. D. B. Collins,  _An Introduction to Regge Theory and High Energy Physics_, Cambridge Monographs on Mathematical Physics (Cambridge University Press, 2023).
  * Gasser and Leutwyler [1984] J. Gasser and H. Leutwyler, Annals of Physics 158, 142 (1984).
  * Rosselet  _et al._ [1977] L. Rosselet, P. Extermann, J. Fischer, O. Guisan, R. Mermod, R. Sachot, A. M. Diamant-Berger, P. Bloch, G. Bunce, B. Devaux, N. Do-Duc, G. Marel, and R. Turlay, Phys. Rev. D 15, 574 (1977).
  * Adeva  _et al._ [2011] B. Adeva _et al._ , Phys. Lett. B 704, 24 (2011), arXiv:1109.0569 [hep-ex] .
  * Liu  _et al._ [2013] X.-H. Liu, F.-K. Guo, and E. Epelbaum, Eur. Phys. J. C 73, 2284 (2013), arXiv:1212.4066 [hep-ph] .
  * Loewe and Zamora [2022] M. Loewe and R. Zamora, Phys. Rev. D 105, 076011 (2022), arXiv:2202.08873 [hep-ph] .
  * Loewe  _et al._ [2022] M. Loewe, D. Valenzuela, and R. Zamora, Phys. Rev. D 105, 036017 (2022), arXiv:2112.13872 [hep-ph] .
  * Blumenson [1960] L. E. Blumenson, The American Mathematical Monthly 67, 63 (1960).
  * Peyaud [2009] B. Peyaud (NA48), Nucl. Phys. B Proc. Suppl. 187, 29 (2009).


ifaamas [AAMAS ’24]Proc. of the 23rd International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2024)May 6 – 10, 2024 Auckland, New
ZealandN. Alechina, V. Dignum, M. Dastani, J.S. Sichman (eds.) 2024 2024 453
Carnegie Mellon University, Meta Pittsburgh Pennsylvania USA Meta Pittsburgh
Pennsylvania USA Carnegie Mellon University, Meta Pittsburgh Pennsylvania USA
Carnegie Mellon University Pittsburgh Pennsylvania USA

# Bootstrapping Linear Models for Fast Online Adaptation in Human-Agent
Collaboration

Benjamin A. Newman<EMAIL_ADDRESS>,  Chris Paxton<EMAIL_ADDRESS>,  Kris
Kitani<EMAIL_ADDRESS>and  Henny Admoni<EMAIL_ADDRESS>
###### Abstract.

Agents that assist people need to have well-initialized policies that can
adapt quickly to align with their partners’ reward functions. Initializing
policies to maximize performance with unknown partners can be achieved by
bootstrapping nonlinear models using imitation learning over large, offline
datasets. Such policies can require prohibitive computation to fine-tune in-
situ and therefore may miss critical run-time information about a partner’s
reward function as expressed through their immediate behavior. In contrast,
online logistic regression using low-capacity models performs rapid inference
and fine-tuning updates and thus can make effective use of immediate in-task
behavior for reward function alignment. However, these low-capacity models
cannot be bootstrapped as effectively by offline datasets and thus have poor
initializations. We propose BLR-HAC, Bootstrapped Logistic Regression for
Human Agent Collaboration, which bootstraps large nonlinear models to learn
the parameters of a low-capacity model which then uses online logistic
regression for updates during collaboration. We test BLR-HAC in a simulated
surface rearrangement task and demonstrate that it achieves higher zero-shot
accuracy than shallow methods and takes far less computation to adapt online
while still achieving similar performance to fine-tuned, large nonlinear
models. For code, please see our project page
https://sites.google.com/view/blr-hac

###### Key words and phrases:

Assistive Robotics; Online Assistance; Human-Robot Interaction; Collaborative
Assistance

##  1\. Introduction

Agents that collaborate with people to complete a person’s preferred goal
cannot always know this preference in advance of an interaction. Though people
may initially state these preferences, they may drift, sometimes changing
entirely, over the course of multiple interaction episodes. While there may be
no continued explicit communication between collaborative partners, people’s
in-situ behaviors are goal-driven and thus can reveal the up-to-date
preference. This means that updating agent policies based on in-situ behaviors
is critical for assisting people during collaborations, i.e. ensuring that
robot actions are deferential to user goals (Newman et al., 2022a).

Figure 1. One step of an example surface rearrangement task: cupboard
organization. From left to right: a person (H) picks an object to place in the
dishwasher; the agent (A) initially places this incorrectly; the person
corrects the placement. From this, the agent learns that the user likes to
place blue objects on the bottom shelf and can place the next, similar object
correctly.

A diagram showing one step of an example surface rearrangement task: cupboard
organization. From left to right: a person (H) picks an object to place in the
dishwasher; the agent (A) initially places this incorrectly; the person
corrects the placement. From this, the robot learns that the user likes to
place blue objects on the bottom shelf and can place the next, similar object
correctly.

Much current research in human-agent collaboration aims to learn zero-shot
collaboration policies from offline datasets that are either collected from
human-human demonstrations Carroll et al. (2019) or generated synthetically
Strouse et al. (2021). Instead of using an individual’s in-situ behavior to
update a model online to improve performance with respect to that individual’s
preference, these approaches train agents offline in collaboration with the
population of partner agents represented by the training dataset. They then
target good performance in aggregate on task metrics. At test time, these
approaches assume the preferences and behavior of a new human collaborator
will fall within the distribution of the collaborators represented by the
training data. While these approaches have been shown to be effective on task
metrics in general collaboration settings, they do not necessarily transfer to
the stricter criteria of assistive collaborations where success in a task is
dictated by a personal preference and people’s goals and behaviors can drift
away from the training distribution.

Furthermore, the population of personal preferences is substantial and
diverse, making it difficult to ensure sufficient coverage during training
time. Collecting large datasets of human-human data is time-consuming and
expensive, while collaboration among populations of procedurally generated
agents can yield data that do not tightly match the distribution of the human
population. Furthermore, as people repeatedly execute a collaborative task,
they may develop new preferences that are unlikely to be captured by the
distribution of collaboration data represented in offline datasets.

We propose a method that takes advantage of these advancements in zero-shot
coordination and applies them to algorithms for fast, online adaptation from
in-situ behavior. In this way, we hope to achieve both good initial
performance when assisting a new partner, but also to continue to adapt to
their preference over continued exposure.

Deciphering people’s exact preferences can be difficult, however, as these
preferences are often not explicitly stated and can change over the course of
an interaction. Fortunately, in-situ behaviors are goal-directed and can
implicitly reveal information about a person’s current preference or goal,
even when it is not expressly communicated. We suggest that agents engaging in
assistive collaborations utilize these goal-directed behaviors to infer and
act towards a person’s current goal, thereby enabling personalized assistance.

To do develop a model that can utilize these goal-directed behaviors for
collaborative assistance, we introduce BLR-HAC: Bootstrapped Logistic
Regression for Human Agent Collaboration. This model is trained using a two-
stage approach: first, we pretrain a transformer Vaswani et al. (2017) to
learn to produce the parameters of a shallow, parameterized policy that
second, is updated throughout a human-agent collaboration using online
logistic regression. To test BLR-HAC, we first introduce a formalization of a
specific instance of a rearrangement task, which we call assistive surface
rearrangement. We then compare BLR-HAC’s performance in a simulated version of
this task against two baselines: 1) a traditional transformer trained with
behavior cloning and 2) a traditional shallow policy trained with online
logistic regression.

Our chosen domain of surface rearrangement models household tasks, like
dishwasher loading, which have complex, long-term dependencies determined by a
combination of a person’s environment and their strongly held preferences. For
example, a person may prefer to place large dishes before small ones to
maximize capacity. Such high dimensional state and preference spaces lead to
an almost infinite number of diverse and equally valid solutions for
completing any given household chores. For example, choosing to load a
dishwasher based on dish material is just as valid as loading based on dish
size; it is a matter of personal preference. Given this diversity, household
tasks make especially good testbeds for studying algorithms that require
aligning robot policies with people’s reward functions, thus mimicking many
use cases for assistive robotics.

While some prior approaches to developing autonomous assistants for household
tasks rely on people providing full task demonstrations in advance of a
collaboration, BLR-HAC aims to operate in real time, utilizing information
from each action as it is taken by a person. Furthermore, approaches relying
on full task demonstrations can introduce additional burden on a person and be
redundant to the goal-directed behavior people exhibit when completing tasks
Baker et al. (2007). In contrast, training shallow, low-capacity models with
logistic regression through MaxEntIRL to utilize in-situ behavior has been
shown to effectively and quickly adapt to people’s objectives in areas such as
robot teleoperation Javdani et al. (2018) and motion planning Losey et al.
(2022).

We test BLR-HAC in a simulated version of our surface rearrangement task. We
find that BLR-HAC outperforms baseline low-capacity models and large,
nonlinear models trained with behavior cloning in zero-shot coordination. We
also find that BLR-HAC achieves similar performance but requires a fraction of
the compute of a transformer that is fine-tuned online. This finding holds
true when considering both preferences that remain the same over time, i.e.
are stationary, and those that drift, i.e. are nonstationary. Taken together,
these results show how BLR-HAC is able to take advantage of the strengths of
both zero-shot and fast online adaptation methods. It does this by pretraining
a large, nonlinear model to learn the parameters of a shallow policy that can
be updated with online logistic regression. This results in a collaborative
agent that is both well-initialized and highly adaptable.

In this paper we make the following contributions:

  * •

a formalization of common household tasks as collaborative IRL tasks, which we
call surface rearrangement,

  * •

a novel model, BLR-HAC, that combines the strengths of pretrained large,
nonlinear models with low-capacity models trained online via logistic
regression for efficient learning in human-robot collaborations, and

  * •

evidence from experiments in simulation that BLR-HAC outperforms its component
models.

##  2\. Related Work

First, we present work in state and action-conditioned models that do not
explicitly learn about their human partner during task execution. We follow
this by reviewing work in adaptive collaborations.

###  2.1. State and Action-Conditioned Collaboration

Prior approaches to solving long-horizon tasks with complex temporal
dependencies and under specified solutions, such as those present in our
surface rearrangement domain, can rely on resolving ambiguities through a
combination of teleoperation and pre-programmed routines Ciocarlie et al.
(2012), or by suggesting optimal, predetermined solutions Newman et al.
(2020b). Solutions following the former method can place undue burden on a
person to explicitly express their preferences, and render robot action
redundant when a demonstration completes. They also require people to
continually demonstrate their desired solution as the constraints of the task,
such as a person’s preference or the environment, vary. Methods following the
latter example do not allow for full freedom of expression from the user and
assume all users have the same “optimal” solution.

Zero-shot coordination is a recent field of research aiming to develop models
that can successfully and immediately interact with novel partners. This can
be done by pretraining models in simulation against agents designed to mimic
human behavior Carroll et al. (2019) or over a diverse population of simulated
agents Strouse et al. (2021). Using these methods, though, can lead to overly
specific solutions. Others have used large language models trained with web-
scale data to propose task plans that are then executed by robots Ahn et al.
(2022). These task plans are not adapted to an individual user, whose reward
function may or may not fit well within the distribution seen during training.
These methods place the burden on the person to either accept a less preferred
robot behavior or continue to provide actions that increase the likelihood of
the robot behavior exhibiting behavior in line with the person’s preferences.
In this work, we focus on combining these good initializations with online
adaptation.

Often, approaches relying solely on large, pretrained deep neural networks
require people to generate explicit descriptions of their preferences which
can be decoded by the model into robot action Ahn et al. (2022). Actions
produced from this process are not guaranteed to align with a person’s task
objective. While deep networks can potentially be adapted to meet individual
preferences through fine-tuning Chen et al. (2022a); He et al. (2022), doing
so with large models can lead to challenging, unstable learning that results
in variable performance Mosbach et al. (2021). In this work, we focus on
developing an algorithm that can quickly adapt to people’s naturally
expressed, task-oriented behavior.

###  2.2. Adaptive Collaborations

Using IRL for robot control can be difficult, in part, due to the ambiguity
that arises from traditional IRL Abbeel and Ng (2004). Maximum entropy IRL
facilitates this by using the principle of maximum entropy to order solutions
according to how well they match observed user behavior Ziebart et al. (2008).
This solution has also been used in behavioral science to model people’s
ability to infer others’ goals from their behavior as exhibited during goal-
directed plans Baker et al. (2007).

These insights have been applied to robot trajectory optimization for shared
control. In the difficult task of teleoperating a high-degree of freedom robot
arm with a low-degree of freedom input device, such as a joystick, a robot can
observe user input commands and infer the user’s most likely goal from a set
of predetermined goals. The robot then assists the user by moving along a path
towards the predicted goal Javdani et al. (2018). MaxEntIRL can also be used
to interpret less direct forms of user behavior, such as physically pushing a
robot out of the way to determine which path the user prefers the robot to
take, for example to carry a coffee mug around a laptop computer instead of
over it Losey et al. (2022), using naturalistic eye gaze in combination with
joystick signals to control a robot arm Newman et al. (2022b); Aronson et al.
(2018); Aronson and Admoni (2022), or using corrective actions to learn about
features of the environment that relate to a person’s preference to increase
generalizability and sample efficiency Newman et al. (2023). We are interested
in adapting online MaxEntIRL for determining high-level task plans consistent
with user preferences in household collaborations from in-task corrective
behavior.

IRL has also been applied to learn robot policies in other types of human-
robot interactions. For example, to learn people’s preferences from
observations of independent task demonstrations Woodworth et al. (2018), or by
learning assistive social actions for therapy by combining a therapists’
expertise with expert demonstrations Andriella et al. (2022), or for social
health, such as a robot receptionist learning to give hygiene advice in a
shopping mall Chen et al. (2022b). Our formulation learns preferences from in-
situ, collaborative behavior for collaborative rearrangement tasks.

Finally, another important aspect of maintaining assistive human-agent
collaborations is to maintain collaborative fluency Hoffman (2019).
Maintaining principles of collaborative fluency, such as minimizing agent and
human idle time, allows human-agent collaborations to function similarly to
human-human collaborations, thereby reducing friction on people to interact
with autonomous agents. Furthermore, robots assisting people to complete
collaborative tasks has been shown to affect a person’s ultimate decision
Newman et al. (2020a), making it important to continually monitor and assess
people’s goals during collaboration. In this work, we will use these ideas as
justification for our desire to develop an algorithm that adapts to user
preferences in real-time.

##  3\. Methods

We formalize the task of surface rearrangement, a specific instance of
rearrangement problems Batra et al. (2020); Szot et al. (2021), as a
decentralized partially observable Markov decision problem (DEC-POMDP).

###  3.1. Defining Surface Rearrangement

Algorithm 1 Surface Rearrangement

1:$\pi_{\theta},\pi_{\hat{\theta}}$, env, $O,L$

2:$s^{0}\leftarrow\text{env.reset}\left(\right)$

3:$\xi\leftarrow\left[\ s^{0}\ \right]$

4:while $\xi\text{.length}<L$ do

5: $a_{h}^{t}\leftarrow\pi_{\theta}\left(\cdot|s^{t-1}\right)$

6: $a_{r}^{t}\leftarrow\pi_{\hat{\theta}}\left(\cdot|a_{h}^{t},s^{t-1}\right)$

7:
$s^{t},a_{c}^{t}\leftarrow$env.step$\left(a_{h}^{t},a_{r}^{t},s^{t-1}\right)$

8: $\xi.$append$\left(\left[a_{h}^{t},a_{r}^{t},a_{c}^{t},s^{t}\right]\right)$

9:end while

To study assistive collaborations, we introduce assistive surface
rearrangement, a collaborative pick and place task where two agents work
together to arrange a set of objects $O$ into a set of locations $L$. In this
task, the assistive agent aims to help a person rearrange objects into
locations. Importantly, the agent’s goal is to achieve the final state that is
desired by the person, which is initially unknown to the agent.

A single episode of this task consists of an object repository containing
objects $o\in O$. The initial state of the episode is $L$ randomly chosen
objects from $O$, and $L$ vacant locations. Each location has a capacity for a
single object. Progress in the task is made by placing objects $o$ into
locations $l\in L$. A task is completed when all objects $o$ have been placed
into a location $l$. For simplicity, we assume that $N\leq|L|$ and that
placing $o$ in $l$ occurs instantaneously.

Two agents interact in an episode in the following way. The human agent
$\pi_{\theta}$ first picks an object given the current state $s^{t-1}$. Then,
the robot agent $\pi_{\hat{\theta}}$ places this object into a location. The
environment then returns the next state $s^{t}$ and the human corrects the
robot’s action, returning $a_{c}^{t}$. An episode $\xi$ can be represented as
the following tuple:
$\left(s^{0},a_{h}^{1},a_{r}^{1},a_{c}^{1},s^{1},...a_{h}^{L},a_{r}^{L},a_{c}^{L},s^{L}\right)$.

###  3.2. Formalizing Surface Rearrangement

Given this description, we can model assistive surface rearrangement as a
decentralized partially observable Markov decision problem (DEC-POMDP) which
is a tuple of $(S,\Pi,A,T,Z,O,r,\gamma)$. Our objective is to train a policy
$\pi_{r}$ that solves this DEC-POMDP:

  * •

S is the set of all possible states. As in prior work Losey et al. (2022), we
assume that a particular state $s\in S$ is a tuple of observable and
unobservable features: $s=(x,\\{\theta_{i}\\})$. Observable state features are
represented as a tuple of all possible locations and all possible objects.
Locations are represented by their ID and their current occupancy. Objects are
represented by their ID and the location they currently occupy, if any. The
unobservable portion of the state, $\theta_{i}$ describes the learnable
parameters of the reward function consistent with the human’s preference in
the task.

  * •

$\Pi$ is the set of agents. In our initial version of this problem, we assume
two agents: a human agent and an assistive agent.

  * •

$A_{i}$ is the set of actions for a particular agent $a_{i}$. We assume that
the person both selects objects and corrects object placements, while the
robot can only make object placements.

  * •

$Z_{i}$ is the set of observations used to infer $\theta$. The assistive
agent’s observation space is the person’s action space. In this work we assume
that the human does not infer the robot’s preference.

  * •

$T(s^{t-1},\mathbf{a}^{t-1},s^{t})$ denotes the transition dynamics that model
the probability of entering a particular state given the current state and
both agents’ actions. As in prior work Losey et al. (2022), changes in $T$ are
dictated by $\theta$. We assume this to be constant and deterministic within a
single episode.

  * •

$O_{i}(s^{t+1},u^{t}_{i},z^{t+1})$, the observation distribution for agent
$\pi_{i}$.

  * •

$r_{i}(s^{t},\\{a_{i}\\}^{t})$ is the reward function for the each agent. We
assume an assistive setting where the agent is trying to estimate and maximize
the person’s reward function. We therefore assume all agents have the same
reward function.

  * •

$\gamma$, a discounting factor.

Given that we assume two agents and that we are only optimizing one (because
the other is assumed to be a person over whose policy we have no control),
this problem reduces to a single agent problem, allowing it to be decomposed
to a POMDP. Since POMDPs are computationally intractable to solve exactly, we
use the QMDP approximation Littman et al. (1995). Prior work in online human
robot collaboration Losey et al. (2022) has shown how a QMDP can be solved
online using online gradient descent, adapted for our purpose in Alg. 3.

##  4\. Approach

Our ultimate goal is to learn an assistive policy that collaborates with a
person during a surface rearrangement task. Given that we want our policy to
be assistive, it should take actions that are aligned with the person’s
underlying preference for completing the task. We interpret this as a regret
minimization problem, where the policy aims to minimize the regret of its
actions with respect to the actions that would be exhibited under the person’s
true preference for completing the task. Importantly, we assume that the
policy does not have prior knowledge of this preference and that the person
does not immediately or explicitly reveal it. Additionally, we assume that the
space of possible preferences the person could hold to be extremely large,
making disambiguation from limited interaction with the person difficult.

Under these conditions, we have two main ways to perform regret minimization.
First, we can ensure our policy takes good initial actions that are likely to
align with the person’s preference, often referred to as zero-shot
performance. Second, we can adapt the policy online as a history of behavior
is accumulated.

Action inference and policy adaptation do not operate within a vacuum, but
rather within the course of the interaction. The common metric in human-robot
interaction of collaborative fluency Hoffman (2019), for example, is critical
to people considering an interaction with a robot to be “good.” An important
facet of this metric is related to the amount of time the robot sits idle
during task execution. This makes frequently updating large models during an
interaction challenging, as both action inference and policy updating require
large amounts of computation, leading to high robot idle times. We aim to
develop a method that can take advantage of the good performance of large
nonlinear models while being able to quickly adapt to user preferences, as
expressed through their in-situ behavior, without causing the robot to idle.

To learn an assistive policy that solves the DEC-POMDP discussed in Sec. 3.2,
we first generate a simulated dataset of diverse, high-level preferences (Sec.
4.1.1). Using these preferences, we collect a dataset of collaborative
demonstrations in a simulated surface rearrangement task over a range of
difficulties (Sec. 4.1.2). We then train our two-stage algorithm by first,
learning to mimic the collected expert demonstrations (Sec. 4.2) and second,
using the preference representations learned in Sec. 4.3 to perform fast,
online adaptation.

###  4.1. Datasets

Figure 2. BLR-HAC Overview From left to right, we first embed the input state
and actions using $\phi$. These are then concatenated and fed into the
preference estimator $\psi$. This learns to output reward parameters, $\theta$
which are used to initialize an online learning policy using the policy $\pi$,
which determines the robot’s action $a_{r}$.

A system diagram of the model we train. We start on the left with the inputs,
which are the human action, the robot action, and the current state. Each of
these is fed through a parameterized feature function specific to each input.
These are then concatenated and fed through a transformer block, which outputs
a preference representation θ. This is used as the parameters for a policy π,
which outputs a robot action for the current timestep.

####  4.1.1. Modeling a Diverse User Population

The two key ideas of our method to develop assistive robots for household
collaborations is that the method should be able to both effectively use a
large population of preference data to pretrain good initializations and be
able to quickly adapt to a particular preference when presented with
information about that preference.

To capture these ideas in our experiments, we develop a simulated dataset of
preferences. First, we sample a large set of preferences, representing a
population, as encoded by $\theta$. We assume preferences from within this
population are drawn normally from one of several modes, each of which
indicates a subpopulation of similar preferences. We sample three preference
datasets: train, , and test. From each set of preferences, we sample episodes
of surface rearrangement episode rollouts, thus creating three datasets:
$D_{train},\ D_{eval},\ \text{and}\ D_{test}$. $D_{train}$ consists of 1000
simulated preferences, sampled from four modes, with 1000 episodes per
preference. $D_{eval},\ \text{and}\ D_{test}$ each contain 100 simulated
preferences, with 20 episodes per preference.

####  4.1.2. Environments for Surface Rearrangement

To test the efficacy of our approach at varying difficulties, we develop three
environments. Each environment scales problem difficulty by increasing the
size of the state space. We have a small environment, with five possible
objects and five locations, a medium environment, with ten objects and ten
locations, and finally a large environment, with 25 objects and 25 locations.

Algorithm 2 Expert Demonstration Collection

1:$\Theta,\pi,\text{{env}},O,L$

2:$D=\left[\ \right]$

3:for $\theta\ \text{{in}}\ \Theta$ do

4: $\xi\leftarrow$ surfaceRearrangement$\left(\pi_{\theta},\text{{env}, O,
L}\right)$

5: $D\text{{.append}}\left(\xi\right)$

6:end for

To collect a demonstration dataset for each environment, we use Alg. 2.
Importantly, to collect expert demonstrations, we set $\theta=\hat{\theta}$
and use a linear policy $\pi=\phi_{h}(a_{h})\cdot\theta\cdot\phi_{r}(A_{r})$,
where all $\phi$ are implemented as one-hot embedding layers. For each
environment we collect 100 demonstrations from each preference generated in
Sec. 4.1.1.

###  4.2. Learning Preferences in a Diverse User Population

The first step of our proposed algorithm aims to minimize regret by achieving
good zero-shot performance. Ultimately, we want to model $p(a_{r}|s,a_{h})$.
This problem, however, is ill-posed, as two policies parameterized by
different preferences will correctly take two different actions $a_{r}$ given
the same state and human action. To account for this ambiguity, we include a
history of $k$ prior state and action pairs taken under the current preference
and maximize $p(a_{r}|s^{t-k-2:t-1},a_{h}^{t-k-1:t},a_{c}^{t-k-2:t-1})$. For
the sake of brevity, we will slightly abuse notation and refer to this
distribution as $p(a_{r}|s,a_{h},k)$.

Algorithm 3 Learning Priors for Online Linear Regression

1:$D,\mathcal{M},\phi_{h},\phi_{r},\phi_{s}$

2:while training do

3: for $\left(s,a_{h},k\right)$ in $D$ do

4:
$\hat{\theta}\leftarrow\mathcal{M}\left(\phi_{s}\left(s\right),\phi_{h}\left(a_{h}\right),\phi_{r}\left(a_{c}\right)\right)$

5: $a_{r}\leftarrow\operatorname*{arg\,max}_{a_{r}\in
A_{r}}\phi_{h}(a_{h}^{t})\cdot\hat{\theta}\cdot\phi_{r}(A_{r})$

6: loss$\leftarrow p(a_{c})\log q(a_{r})$

7: training $\leftarrow\mathcal{M}.\text{{update}}(\text{loss})$

8: end for

9:end while

Again, when training assistive agents, achieving low zero-shot performance is
not our only objective. We also need an agent that adapts online to incoming
user behavior while maintaining collaborative fluency. This means developing a
lightweight, low-parameter model capable of performing action inference and
policy adaptation in real time.

To do this, instead of learning $p(a_{r}|s,a_{h},k)$ directly, we first learn
a latent space that corresponds to the weights of a logistic regression
problem. These weights serve as the input to the second step of our algorithm,
Sec. 4.3. Thus, we train our model to maximize
$M_{\phi,\psi}(s,a_{h},k,t)=p(\theta|s,a_{h},k,t)$. In this way, we place an
inductive bias over the latent space of the model, enticing it to learn a
matrix of size $O\times L$, that can be used as the weights of an online
logistic regression problem. We treat this as a classification problem and
minimize the cross entropy loss between our model’s predictions and the
collected expert demonstrations: $L=p(a_{c})\cdot\log q(a_{r})$ where
$q(a_{r})=\phi(a_{h})\cdot M(s,a_{h},k,t)\cdot\phi(A_{r})$, as shown in Alg.
3.

###  4.3. Bootstrapping Shallow Linear Models for Fast, Online Adaptation

The second step of our proposed algorithm aims to minimize regret through
online adaptation. Using the output of the model learned in Sec. 4.2, we can
employ online logistic regression, which has been shown to work well for
teaching human preferences to agents through corrective feedback in robot
control tasks. Importantly, since online logistic regression has a very simple
update rule to estimate $\theta$ that operates over a much smaller number of
parameters than a large, nonlinear network, we can adapt this initial estimate
of the person’s preference in-situ without risking large human or robot idle
time, thereby maintaining collaborative fluency.

To update our estimate of $\theta$, we use a linear approximation of the QMDP
solution to the DEC-POMDP in Section 3.2 and stochastic gradient descent,
resulting in the following update rule:

$\hat{\theta}=\hat{\theta}-\alpha\left(\phi_{h}(a_{h})\cdot\phi_{r}(a_{r})-\phi_{h}(a_{h})\cdot\phi_{r}(a_{c})\right)$

where $\alpha$ is the learning rate.

| Small | Medium | Large
---|---|---|---
| No Prior | Prior (ours) | No Prior | Prior (ours) | No Prior | Prior (ours)
ShallowLinear | 0.413 | 0.665 | 0.215 | 0.518 | 0.096 | 0.289
DeepLinear | 0.425 | 0.680 | 0.199 | 0.504 | 0.101 | 0.303
MLP | 0.605 | 0.759 | 0.361 | 0.653 | 0.120 | 0.358
Transformer | 0.729 | 0.771 | 0.603 | 0.673 | 0.160 | 0.412
Table 1. We compare zero-shot performance on the test set of each environment.
We have two axes of comparison: model complexity in the rows, and inductive
prior in the columns. Results are reported in terms of accuracy. We can see
that the highest capacity, attention based model trained with an inductive
prior outperforms all other models in every environment.

##  5\. Experimental Design

To test our algorithm, we design several experiments. First, we validate the
need for large, nonlinear models to learn the distribution of preferences
embedded in the demonstration dataset, Sec. 5.1. Then we explore how our
algorithm fares in its intended use case: fast, online adaptation. We test
this in two scenarios. Sec. 5.2.1 analyzes adaptation to a single preference
over time, while Sec. 5.2.2 explores how well our algorithm fares when the
preference generating the behavior changes without explicit communication to
the robot.

###  5.1. Zero-Shot Coordination

We evaluate our model in each environment over the test set using Alg. 3.
While we are in search of an algorithm that performs regret minimization, this
metric is relative to a specific preference. To understand model performance
in an absolute sense and compare across environments, we report accuracy in
terms of the number of correct robot action predictions. This metric is
inversely correlated with regret.

We choose our baselines to examine two key questions: 1) are high-capacity,
nonlinear models necessary for disambiguation between preferences in a highly
diverse preference space, and 2) how does inducing an inductive prior over the
latent space affect zero-shot performance?

To answer these questions we introduce baselines across two axes: model
complexity and model bias. To determine the effect of high-capacity nonlinear
models on zero-shot performance we compare four levels of model complexity in
terms of how we implement $\psi$ in Fig. 2:

  * •

ShallowLinear. Typical online IRL settings learn a shallow model from scratch
using MaxEntIRL. To bootstrap this process, one could perform the same process
over the offline dataset, thereby encoding the diverse preference population
in the initial model weights. Our intuition, though, is that since
demonstrations are drawn from a large, diverse population of preferences, and
that the relations between preferences and people are not known a priori, this
disambiguation will benefit from a nonlinear function approximator. We expect
nonlinear, high-capacity models to outperform this baseline.

  * •

DeepLinear. Since the space of preferences is very large, it could simply be
that increasing model capacity without introducing nonlinearity may capture
the preference distribution. To test this, we introduce DeepLinear, which
simply adds additional model parameters in both width and depth. We expect
this model to outperform a ShallowLinear model but underperform nonlinear
methods.

  * •

Multi-Layer Perceptron. To test the importance of modeling the preference
distribution with a nonlinear model, we introduce a multi-layer perceptron
baseline. We expect this model to outperform both linear methods but
underperform attention-based mechanisms.

  * •

Causal Transformer. Finally, since we are passing a history of behavior to the
model at every time step, we can infer the current preference from this
sequence of behaviors. Attention-based mechanisms, specifically causal
transformers, have been shown to excel at modeling sequential data. To test
this we implement $\psi$ as a transformer, and expect it to outperform all
other methods.

The second axis of baselines we develop compares the importance of introducing
an inductive bias over the latent space in order to learn $\theta$. We compare
an implementation of the above models in which each model minimizes
$L=p(a_{r})\cdot\log\mathcal{M}(s,a_{h},k)$ to our proposed inductive bias,
which minimizes
$L=p(ar)\cdot\log\phi(a_{h})\cdot\mathcal{M}(s,a_{h},k)\cdot\phi(A_{r})$.

####  5.1.1. Implementation Details

To implement our models we make the following decisions. We perform a separate
parameter sweep for each model and environment for the following parameters
and ranges: learning rate $(1e^{-3},1e^{-6})$, the dimensionality of hidden
layers $(2^{5},2^{8})$, and the number of layers in $\psi$ $(3,5,7,10,12)$. We
set the size of the input history to be 50, padding when necessary. For each
model, we implement all $\phi$ as a single, one-hot embedding space of
vocabulary size 208, where 0-7 are special characters, 8-107 are location
indices, and 108-207 are object indices. To implement $\psi$, we use PyTorch
Paszke et al. (2019) and base our implementation of a causal transformer on
Decision Transformer Chen et al. (2021). We implement $\pi$ as a simple linear
model for inductive bias and as an MLP for no inductive bias. All models are
trained using the appropriate training and evaluation sets, which do not
overlap with the test set, with 10 epochs of early stopping.

###  5.2. Test-Time Adaptation

Developing assistive policies is not only about achieving good zero-shot
performance, however. The space of actual human preferences is almost
boundless and likely impossible to capture in advance of an interaction.
Therefore, it is important to develop algorithms that can rapidly align
themselves with preferences associated with a person’s in-situ behavior. We
study this in two settings. First, we analyze our algorithm’s ability to adapt
to a stationary preference over the course of multiple episodes. Then, we
analyze our algorithm’s ability to adapt in scenarios where preferences are
nonstationary. Here, we are interested in an algorithm’s ability to 1)
maintain decent performance in the face of the preference change, and 2)
rapidly recover after the change in preference.

####  5.2.1. Stationary Preferences

To test our algorithm’s ability to adapt to stationary preferences, we average
the performance of our bootstrapped online IRL algorithm over all preferences
in the testing set over 20 episodes in each testing environment.

We compare against a linear model that learns from scratch and a method that
optimizes over all transformer parameters between episodes but keeps inference
computation constant. We measure computation cost in terms of FLOPS and
calculate these values empirically using FVCore. We expect to see that the
bootstrapped online IRL algorithm achieves similar performance to the online
transformer method but at a fraction of the compute.

####  5.2.2. Nonstationary Preferences

Similar to the stationary preferences experiment, we run IRL over 20 episodes.
In this analysis, however, we switch to a different random objective after 10
episodes. Again, we compare against a linear model learning from scratch and
an online transformer implementation. We expect to see that the linear method
starts with poor performance but adapts quickly when exposed to incoming
behavior. We expect to see that the transformer method starts with good
performance and adapts more slowly as behavior data is accumulated. Finally,
we expect our method to achieve the benefits of both the linear from scratch
and the transformer methods: it should start off with reasonable performance
and adapt quickly as data is aggregated.

####  5.2.3. Implementation Details

For both experiments, we do a hyperparameter sweep over the learning rate in
the range $(1e^{-2},1e^{-5})$ for the transformer and $(1,5,10)$ for the
linear models. In both cases, we use the maximum learning rate for all
experiments. Additionally, we use stochastic gradient descent for optimization
in both cases. To train the transformer method, we perform five steps of
gradient descent between each episode.

15101520$0$$0.2$$0.4$$0.6$$0.8$$1$Episode NumberAccuracy (% correct)Stationary
Adaptation in $\displaystyle Small$LinearBLR-HACTransf

A graph showing the empirical results for the stationary adaptation
experiments in the medium environment. The graph layout is the same as for the
small environment, but this time the red line starts much lower than the other
two. Additionally, all lines start lower than in the small environment and end
at the same place, but lower than in the small environment.

15101520$0$$0.2$$0.4$$0.6$$0.8$$1$Episode NumberStationary Adaptation in
$\displaystyle Medium$

A graph showing the empirical results for the stationary adaptation
experiments in the medium environment. The graph layout is the same as for the
small environment, but this time the red line starts much lower than the other
two. Additionally, all lines start lower than in the small environment and end
at the same place, but lower than in the small environment.

15101520$0$$0.2$$0.4$$0.6$$0.8$$1$Episode NumberStationary Adaptation in
$\displaystyle Large$

A graph showing the empirical results for the stationary adaptation
experiments in the large environment. The graph layout is the same as for the
small and medium environment, the red line again starts much lower than the
other two. Additionally, all lines start lower than in the small and medium
environments and end at the same place, but lower than in the small and medium
environments.

Figure 3. Stationary Test-Time Adaptation. Learning curves for each test
environment for each algorithm. We report the average accuracy over each
episode. BLR-HAC is able to achieve the low zero-shot performance of the
transformer method, and the fast adaptation of the linear method.
Additionally, we can see that as the episode length increases, these
differences in performance are more notable, with the linear method failing to
catch up to the other two methods over the course of 20 episodes.

15101520$0$$0.2$$0.4$$0.6$$0.8$$1$Episode NumberAccuracy (%
correct)Nonstationary Adaptation in $\displaystyle Small$LinearBLR-HACTransf

A graph showing the empirical results for the nonstationary adaptation
experiments in the small environment. There are three smooth lines, one in red
for the linear method, one in green for our method, and one in yellow for the
transformer method. There is a sharp decline at step 10 on the x-axis. All
lines start at roughly the same position on the y-axis (at around 0.75) and
recover after the decline to approximately 1 at the 20th time step.

15101520$0$$0.2$$0.4$$0.6$$0.8$$1$Episode NumberNonstationary Adaptation in
$\displaystyle Medium$

A graph showing the empirical results for the nonstationary adaptation
experiments in the medium environment. The graph layout is the same as for the
small environment, but this time the red line starts much lower than the other
two. Additionally, all lines start lower than in the small environment and end
at the same place, but lower than in the small environment.

15101520$0$$0.2$$0.4$$0.6$$0.8$$1$Episode NumberNonstationary Adaptation in
$\displaystyle Large$

A graph showing the empirical results for the nonstationary adaptation
experiments in the large environment. The graph layout is the same as for the
small and medium environment, the red line again starts much lower than the
other two. Additionally, all lines start lower than in the small and medium
environments and end at the same place, but lower than in the small and medium
environments. There is a larger separation at the 20th timestep between all
three lines (ranked yellow then green then red) than in either of the other
two experiments.

Figure 4. Nonstationary Test-Time Adaptation. Learning curves over each test
environment for each algorithm. We report average accuracy over episodes. BLR-
HAC is able to perform on par with the transformer method in the small and
medium environments and part of the large environment. BLR-HAC outperforms all
methods in all environments immediately after the preference switch. In the
large environment, though, the transformer recovers more quickly as it has
access to more data.

##  6\. Results

From running the experiments outlined in Sec. 5, we have three main results.
First, we find support for our hypothesis that nonlinear, high-capacity models
trained with inductive biases can learn a diverse population of user
preferences. In Tab. 1, we see the attention-based method trained with an
inductive prior outperforms all other methods, achieving
$77.1\%,67.3\%,\text{and}\ 41.2\%$ accuracy on the small, medium, and large
environments, respectively. We see that the difference in performance between
models trained with and without the inductive prior increases as the
difficulty of the problem increases. Additionally, we see the general trend
that higher capacity, nonlinear models outperform lower capacity linear
models. These results empirically justify our desire to use a high-capacity
nonlinear model to bootstrap a linear model in an online logistic regression
problem.

Our second set of results is shown in Fig. 3. Here, we plot the test-time
adaptation accuracy for three models: linear (in red), BLR-HAC (in green), and
an online transformer (in yellow). From these graphs, we can see support for
our hypothesis that bootstrapped, shallow linear models trained with IRL
achieve good accuracy with low computation. We can see that BLR-HAC and
Transformer both start with higher accuracy than Linear in all cases and that
this difference increases as the problem complexity increases. Furthermore, we
see how BLR-HAC achieves similar performance over episodes as the transformer
method, but at a fraction of the computation. While both methods have similar
inference compute, of $OxL$ FLOPS, BLR-HAC uses only $2xOxL$ FLOPS, while the
Transformer method uses $\sim 400M$ FLOPS during updates.

Finally, we see in Fig. 4 results from test-time adaption with nonstationary
preferences. These results show mixed support for our hypothesis that
bootstrapped, shallow linear models trained with IRL recover well from
unexpected shifts in user behavior. In each graph, episodes 1-10 show similar
results to the previous set of experiments. At episode 10, however, the
preference shifts, and all models suffer a drop in performance. Interestingly,
in all cases, BLR-HAC suffers the smallest drop in performance. While this is
a positive result, we also see that as the environment becomes more complex,
BLR-HAC suffers in its adaptation rate from episodes 10-20. While it adapts on
par with the linear method (though still achieves higher performance due to
its better initial performance) it adapts slower than the transformer-based
method. This is likely due to the fact that the transformer is able to make
better use of the larger amounts of data that are being aggregated in the
large environment.

##  7\. Discussion, Limitations, and Future Work

We develop policies for assistive agents that are both well-initialized and
highly-adaptable. Through simulated experiments, our method achieves both the
good initializations of large, nonlinear models trained with behavior cloning
and the fast adaptation to user behavior present in low-capacity models
trained with online MaxEntIRL. Importantly, BLR-HAC initializes better than
ShallowLinear on test data that is far from the initial distribution, meaning
that our approach should ideally allow for faster adaptation to populations
for whom it is difficult to collect data for offline pretraining.

Future work should explore applying BLR-HAC to user studies with real people
to determine whether the better initializations and faster adaptations of our
method hold outside of simulation and are preferred. It is also important to
study how the effect of the size of the surface rearrangement problem on these
results.

User studies also provide an opportunity to improve our method. Collecting
interaction data through interactive simulators, such as AI Habitat Savva et
al. (2019); Szot et al. (2021), deployed on platforms such as Amazon
Mechanical Turk Crowston (2012) or Prolific Prolific (line) would allow us to
pretrain BLR-HAC with real data.

Finally, our method also assumes a single, synchronized modality of corrective
actions: direct state corrections. This makes our learning problem easier by
maximizing the correlation between the leader’s corrections and their reward
function. We would like to extend our approach to account for other modalities
of corrections issued asynchronously, such as those expressed in real time
through verbal or nonverbal communication.

##  8\. Conclusion

In this work, we laid out an argument for why assistive agents should be both
well-initialized and highly-adaptable. We introduced a novel formulation of
assistive human-agent collaboration as collaborative inverse reinforcement
learning and introduced an algorithm BLR-HAC that takes advantage of
sophisticated population-level modeling found in deep neural networks with the
fast adaptation of shallow, low-capacity inverse reinforcement learning
methods. Finally, we verified these claims through simulated experiments.

##  9\. Ethics Statement

We show we can use offline datasets to bootstrap assistive collaborations by
pretraining assistive agents. This method, however, necessitates using
specific subpopulations of the larger human population, i.e. those represented
by the dataset. This leads to ethical questions such as: Are the preferences
present in the dataset representative of the larger population? How does this
affect people who hold preferences outside this subpopulation? These questions
are especially pertinent in assistive settings, where agents are likely to
encounter out-of-distribution phenomena at test-time. It is questions such as
these that motivate this work.

We assume a critical part of providing assistance is to reduce unnecessary
burden placed on individuals while acting in alignment with their preference.
When a person’s preferences are well represented by the dataset, pretraining
necessarily minimizes a person’s burden to bring the agent into alignment with
their preference. When a person’s preferences are not well represented by the
dataset, our method aligns to the person’s preference quickly by using their
in-situ, goal-directed behavior. Thus, while the model does not have an
initial representation of these out-of-domain preferences, it does know how to
interpret goal-directed behaviors in order to learn such a representation.

We believe there is ample opportunity for future work to continue to explore
solutions to these ethical dilemmas, such as to learn more generalizable
features of preferences that allow for better representations of human
preferences, or by teaching agents to learn to learn preferences, which would
improve an assistive agents ability to adapt to out-of-distribution
preferences.

## References

  * (1)
  * Abbeel and Ng (2004) Pieter Abbeel and Andrew Y Ng. 2004.  Apprenticeship learning via inverse reinforcement learning. In _Proceedings of the twenty-first international conference on Machine learning_. 1.
  * Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. 2022\.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ (2022).
  * Andriella et al. (2022) Antonio Andriella, Carme Torras, Carla Abdelnour, and Guillem Alenyà. 2022.  Introducing CARESSER: A framework for in situ learning robot social assistance from expert knowledge and demonstrations.  _User Modeling and User-Adapted Interaction_ (03 2022).  https://doi.org/10.1007/s11257-021-09316-5
  * Aronson and Admoni (2022) Reuben M. Aronson and Henny Admoni. 2022.  Gaze Complements Control Input for Goal Prediction During Assisted Teleoperation.  _Robotics science and systems_ (2022).  https://par.nsf.gov/biblio/10327640
  * Aronson et al. (2018) Reuben M. Aronson, Thiago Santini, Thomas C. Kübler, Enkelejda Kasneci, Siddhartha Srinivasa, and Henny Admoni. 2018.  Eye-Hand Behavior in Human-Robot Shared Manipulation. In _Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction_ (Chicago, IL, USA) _(HRI ’18)_. Association for Computing Machinery, New York, NY, USA, 4–13.  https://doi.org/10.1145/3171221.3171287
  * Baker et al. (2007) Chris L Baker, Joshua B Tenenbaum, and Rebecca R Saxe. 2007.  Goal inference as inverse planning. In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , Vol. 29.
  * Batra et al. (2020) Dhruv Batra, Angel X Chang, Sonia Chernova, Andrew J Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. 2020\.  Rearrangement: A challenge for embodied ai.  _arXiv preprint arXiv:2011.01975_ (2020).
  * Carroll et al. (2019) Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019.  On the utility of learning about humans for human-ai coordination.  _Advances in neural information processing systems_ 32 (2019).
  * Chen et al. (2021) Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021.  Decision Transformer: Reinforcement Learning via Sequence Modeling.  arXiv:2106.01345 [cs.LG]
  * Chen et al. (2022a) Sean Chen, Jensen Gao, Siddharth Reddy, Glen Berseth, Anca D. Dragan, and Sergey Levine. 2022a.  ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement Learning. In _2022 International Conference on Robotics and Automation (ICRA)_. 7505–7512.  https://doi.org/10.1109/ICRA46639.2022.9812442
  * Chen et al. (2022b) Zhichao Chen, Yutaka Nakamura, and Hiroshi Ishiguro. 2022b.  Android as a Receptionist in a Shopping Mall Using Inverse Reinforcement Learning.  _IEEE Robotics and Automation Letters_ 7, 3 (2022), 7091–7098.  https://doi.org/10.1109/LRA.2022.3180042
  * Ciocarlie et al. (2012) Matei Ciocarlie, Kaijen Hsiao, Adam Leeper, and David Gossow. 2012.  Mobile manipulation through an assistive home robot. In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_. 5313–5320.  https://doi.org/10.1109/IROS.2012.6385907
  * Crowston (2012) Kevin Crowston. 2012.  Amazon Mechanical Turk: A Research Tool for Organizations and Information Systems Scholars. In _Shaping the Future of ICT Research. Methods and Approaches_ , Anol Bhattacherjee and Brian Fitzgerald (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 210–221.
  * He et al. (2022) Jerry Zhi-Yang He, Zackory Erickson, Daniel S. Brown, Aditi Raghunathan, and Anca Dragan. 2022.  Learning Representations that Enable Generalization in Assistive Tasks. In _6th Annual Conference on Robot Learning_.  https://openreview.net/forum?id=b88HF4vd_ej
  * Hoffman (2019) Guy Hoffman. 2019.  Evaluating Fluency in Human–Robot Collaboration.  _IEEE Transactions on Human-Machine Systems_ 49, 3 (2019), 209–218.  https://doi.org/10.1109/THMS.2019.2904558
  * Javdani et al. (2018) Shervin Javdani, Henny Admoni, Stefania Pellegrinelli, Siddhartha S. Srinivasa, and J. Andrew Bagnell. 2018.  Shared autonomy via hindsight optimization for teleoperation and teaming.  _The International Journal of Robotics Research_ 37, 7 (2018), 717–742.  https://doi.org/10.1177/0278364918776060 arXiv:https://doi.org/10.1177/0278364918776060
  * Littman et al. (1995) Michael L Littman, Anthony R Cassandra, and Leslie Pack Kaelbling. 1995.  Learning policies for partially observable environments: Scaling up.  In _Machine Learning Proceedings 1995_. Elsevier, 362–370.
  * Losey et al. (2022) Dylan P Losey, Andrea Bajcsy, Marcia K O’Malley, and Anca D Dragan. 2022.  Physical interaction as communication: Learning robot objectives online from human corrections.  _The International Journal of Robotics Research_ 41, 1 (2022), 20–44.
  * Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021.  On the Stability of Fine-tuning {BERT}: Misconceptions, Explanations, and Strong Baselines. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=nzpLWnVAyah
  * Newman et al. (2020b) Benjamin Newman, Kevin Carlberg, and Ruta Desai. 2020b.  Optimal Assistance for Object-Rearrangement Tasks in Augmented Reality.  arXiv:2010.07358 [cs.HC]
  * Newman et al. (2022a) Benjamin A. Newman, Reuben M. Aronson, Kris Kitani, and Henny Admoni. 2022a.  Helping People Through Space and Time: Assistance as a Perspective on Human-Robot Interaction.  _Frontiers in Robotics and AI_ 8 (2022).  https://doi.org/10.3389/frobt.2021.720319
  * Newman et al. (2022b) Benjamin A. Newman, Reuben M. Aronson, Siddhartha S. Srinivasa, Kris Kitani, and Henny Admoni. 2022b.  HARMONIC: A multimodal dataset of assistive human–robot collaboration.  _The International Journal of Robotics Research_ 41, 1 (2022), 3–11.  https://doi.org/10.1177/02783649211050677 arXiv:https://doi.org/10.1177/02783649211050677
  * Newman et al. (2020a) Benjamin A. Newman, Abhijat Biswas, Sarthak Ahuja, Siddharth Girdhar, Kris K. Kitani, and Henny Admoni. 2020a.  Examining the Effects of Anticipatory Robot Assistance on Human Decision Making. In _Social Robotics_ , Alan R. Wagner, David Feil-Seifer, Kerstin S. Haring, Silvia Rossi, Thomas Williams, Hongsheng He, and Shuzhi Sam Ge (Eds.). Springer International Publishing, Cham, 590–603.
  * Newman et al. (2023) Benjamin A. Newman, Christopher Jason Paxton, Kris Kitani, and Henny Admoni. 2023.  Towards Online Adaptation for Autonomous Household Assistants. In _Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ (Stockholm, Sweden) _(HRI ’23)_. Association for Computing Machinery, New York, NY, USA, 506–510.  https://doi.org/10.1145/3568294.3580136
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems 32_. Curran Associates, Inc., 8024–8035.  http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * Prolific (line) Prolific. 2014 Online.  Prolific.  https://www.prolific.co
  * Savva et al. (2019) Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. 2019\.  Habitat: A platform for embodied ai research. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 9339–9347.
  * Strouse et al. (2021) DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. 2021.  Collaborating with humans without human data.  _Advances in Neural Information Processing Systems_ 34 (2021), 14502–14515.
  * Szot et al. (2021) Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John M Turner, Noah D Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimír Vondruš, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel X Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. 2021.  Habitat 2.0: Training Home Assistants to Rearrange their Habitat. In _Advances in Neural Information Processing Systems_ , A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (Eds.).  https://openreview.net/forum?id=DPHsCQ8OpA
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need. In _Advances in Neural Information Processing Systems_ , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * Woodworth et al. (2018) Bryce Woodworth, Francesco Ferrari, Teofilo E. Zosa, and Laurel D. Riek. 2018.  Preference Learning in Assistive Robotics: Observational Repeated Inverse Reinforcement Learning. In _Proceedings of the 3rd Machine Learning for Healthcare Conference_ _(Proceedings of Machine Learning Research, Vol. 85)_ , Finale Doshi-Velez, Jim Fackler, Ken Jung, David Kale, Rajesh Ranganath, Byron Wallace, and Jenna Wiens (Eds.). PMLR, 420–439.  https://proceedings.mlr.press/v85/woodworth18a.html
  * Ziebart et al. (2008) Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. 2008.  Maximum Entropy Inverse Reinforcement Learning. In _Proc. AAAI_. 1433–1438.