# Combining imitation and deep reinforcement learning to human-level
performance on a virtual foraging task

Vittorio Giammarino  Division of Systems Engineering, Boston University,
Boston, MA 02446, USA<EMAIL_ADDRESS>Matthew F Dunne  Cognitive
Neuroimaging Center, Boston University, Boston, MA 02215, USA.  Graduate
Program for Neuroscience, Boston University, Boston, MA 02215, USA.  Center
for Systems Neuroscience, Boston University, Boston, MA 02215, USA.  Kylie N
Moore  Cognitive Neuroimaging Center, Boston University, Boston, MA 02215,
USA.  Graduate Program for Neuroscience, Boston University, Boston, MA 02215,
USA.  Center for Systems Neuroscience, Boston University, Boston, MA 02215,
USA.  Michael E Hasselmo  Center for Systems Neuroscience, Boston University,
Boston, MA 02215, USA.  Chantal E Stern  Cognitive Neuroimaging Center, Boston
University, Boston, MA 02215, USA.  Graduate Program for Neuroscience, Boston
University, Boston, MA 02215, USA.  Ioannis Ch. Paschalidis  Division of
Systems Engineering, Boston University, Boston, MA 02446, USA.  Dept. of
Electrical and Computer Engineering, Boston University, Boston, MA 02446, USA.
Dept. of Biomedical Engineering, Boston University, Boston, MA 02215, USA.

###### Abstract

We develop a simple framework to learn bio-inspired foraging policies using
human data. We conduct an experiment where humans are virtually immersed in an
open field foraging environment and are trained to collect the highest amount
of rewards. A Markov Decision Process (MDP) framework is introduced to model
the human decision dynamics. Then, Imitation Learning (IL) based on maximum
likelihood estimation is used to train Neural Networks (NN) that map human
decisions to observed states. The results show that passive imitation
substantially underperforms humans. We further refine the human-inspired
policies via Reinforcement Learning (RL) using the on-policy Proximal Policy
Optimization (PPO) algorithm which shows better stability than other
algorithms and can steadily improve the policies pretrained with IL. We show
that the combination of IL and RL can match human results and that good
performance strongly depends on combining the allocentric information with an
egocentric representation of the environment.

## Introduction

Human beings are exceptional learners: capable of conceiving solutions for
individual problems, generalizing acquired skills to new tasks, exploring new
strategies, and inferring causal relationships. [1, 2, 3, 4] Since the
earliest stage of Machine Learning (ML), the research community has sought to
emulate humans’ learning capacities; and only recently, works which combine
Deep Learning (DL) with Reinforcement Learning (RL), have accomplished
outstanding results in this regard.[5] DL and RL have shown to be
indispensable ingredients to accomplish human-like intelligence in artificial
systems; however, they still require a massive amount of computational
resources and do not show the same level of efficiency compared to human
beings.[6] A viable option to tackle the efficiency issue, again inspired by
human learning,[7, 8] is to leverage human demonstrations by combining DL and
RL with imitation in a procedure known as imitation learning (IL).[9] It is
worth noting that a great number of tasks, such as navigating or exploring
unknown environments, are relatively straightforward for humans and can be
successfully learned in a limited number of trials. On the other hand, this is
often not the case for artificial agents, where the amount and the quality of
the information retrieved, in addition to a sound design of the reward
function and/or a good exploration strategy of the environment, are crucial
for successfully learning from scratch. Hence, artificial agents might benefit
from directly imitating human behavior or, alternatively, from reconstructing
human-inspired reward functions.[10, 11]

In this study, we investigate the potential of learning from humans taking
into account not only performance but also efficiency. We start by collecting
movement data from a series of human participants while they are performing a
virtual foraging task in which the rewards, in the form of coins, are
condensed in clusters throughout the environment. The participants, subject to
time constraints, have to collect the highest number of coins, effectively
trading-off between foraging within a single cluster (exploitation) and
exploration. Humans are initially unaware of the number of clusters and of
their locations but are able to learn the reward distribution throughout the
course of the experiment.

Note that, time-constrained foraging problems occur in several realistic
scenarios, including scientific exploration; where, for instance, a rover
might want to sample chemical or geological features as fast as possible; or
search and rescue operations, where a vessel needs to rescue as many people as
possible.[12, 13] Moreover, these missions are dangerous, and the use of
aerial or ground unmanned vehicles would significantly mitigate any additional
risks. However, without assumptions about the distribution of targets,
classical control techniques are not applicable. Tele-operation is also a
feasible option, but it may be hindered due to unreliable communications, and
this approach does not scale as well as completely autonomous options. For
these reasons, ML techniques supporting full autonomy represent an interesting
alternative solution.[14] Therefore, our main objective is to develop a method
which effectively combines IL with DL and RL and allows for efficient human-
level learning in a foraging task with sparse rewards.

From our experiment, we collect $50$ human trajectories and further process
them to include both allocentric and egocentric information in our model. We
then run IL on each of the trajectories, yielding $50$ policies with different
performance. None of these policies succeed in matching human results.
Finally, we use the imitated policies as initial solutions and further refine
them with RL. By combining the two methods, we outperform the average human
performance and the respective participant from which the agent imitates with
a success rate of 78% and 62%, respectively, while using a reasonable amount
of training steps ($\leq 10^{7}$). We also compare our method with a pure RL
alternative, and show that such an approach remarkably underperforms humans.

In the final part of the work, we test the learned policies for generalization
and robustness in a new scenario with an unknown reward distribution. We are
able to show that the artificial agents quickly adapt to this new scenario and
conjecture that, when combined with allocentric information, the egocentric
representation of the environment plays a key role in enabling learning as
also observed in neurophysiological recordings from rodents.[15] To
empirically test this hypothesis, we rerun the entire set of experiments only
considering allocentric coordinates. We show that in the absence of an
egocentric representation of the environment, RL is unable to further improve
IL and the final performance does not reach the level of human subjects. For
the sake of completeness, we rerun the experiments also considering only the
egocentric representation of the environment. This setup, however, violates
the MDP assumption and, since our agents are not equipped with explicit
memory, the learnt policies underperform those of the other experiments. A
figure illustrating all these experiments for the full set of $50$ human
trajectories is included in the supplementary materials. We conclude that
proper modeling is as crucial as the right algorithmic choices in order to
enhance general and robust learning in artificial agents.

### Related Work and Contribution

We focus on combining IL with RL in order to address the shortcomings of these
two approaches when used individually. IL was initially proposed as a
supervised learning method for faster policy learning.[9, 16] Recent works
have studied the limitation of IL including the covariate shift problem and
its dependency on the quality of the demonstrations.[17, 18, 19]. RL instead
was proposed to enable learning through direct interaction with the
environment.[20] RL combined with DL has achieved outstanding results in
policy learning[5], however, sample inefficiency remains an obstacle for its
deployment in real world scenarios.[21] Recent works have endeavored to
combine IL with RL either to address the limitations of IL[22, 23, 24, 25] or
to improve efficiency in RL. [26, 27, 28, 29, 30]

Another line of research, known as Inverse Reinforcement Learning (IRL),
leverages demonstrations in order to infer a reward function which is then
used for RL in order to recover the demonstrator policy.[31, 32, 33, 34] IRL
has the pros of being immune to the covariate shift problem but the cons of
being as sample inefficient as the used RL algorithm. A unified view of IRL
and IL as an $f$-divergence minimization problem has been recently
proposed[35, 36] and addressed using Generative Adversarial Networks[37] (GAN)
for either IL[35] or reward shaping[38].

We place our work in the Reinforcement Learning with Expert Demonstrations
(RLED) framework, where the RL agent learns in the same environment of the
demonstrator and using the same reward function.[39] Our main contribution is
leveraging a non trivial case study to show how modeling, imitation and
reinforcement, when effectively combined, can lead to human-like performance
in navigation tasks with sparse rewards without requiring a massive amount of
training steps. Note that, this does not mean that RL-only algorithms cannot
achieve human-level performance in this type of tasks, rather that they need a
significantly larger number of steps to do so. Dense reward signals would have
most likely improved RL-only performance but also assumed prior knowledge of
the environment invalidating therefore the comparison with the human agents.
Moreover, we analyze our results for robustness and empirically show a strong
correlation between egocentric representation of the environment and
performance. We reemphasize the importance of the right algorithmic choices as
well as the right model in order to enhance effective learning in artificial
agents.

Two works which likewise combine IL with RL are Deep Q-Learning from
Demonstrations (DQfD)[39] and AlphaGO[5]. However, with respect to AlphaGO,
which lies in the model-based spectrum, our work considers a pure model-free
RL setting. DQfD, on the other hand, explores pretraining a deep Q network[40]
using demonstrations before performing RL. In order to do so, the agent needs
access not only to the demonstrator state-action pairs but also to the rewards
collected along the trajectories. In other words, DQfD assumes access to the
full MDP transition (states, actions and rewards) and as a result, its
pretrain step can be seen as a first form of offline RL [41]. In our study, we
assume access only to demonstrator state-action pairs (without rewards) and
therefore DQfD is not directly applicable. Furthermore, none of the
aforementioned works investigate the effects of modeling on algorithmic
performance.

### Outline and Notation

The remainder of the paper is organized as follows. The Materials and Methods
section presents the experimental setup used to collect the human foraging
data, introduces the MDP model used for representing human behavior, discusses
the IL for learning policies from data, and outlines the RL algorithms used to
refine the imitated policies. In the Results section we compare our method
with human and RL-only performance on the original setup; then, we test all
the artificial agents for robustness to reward distribution shift and
demonstrate the importance of egocentric information. We discuss the results
in the Discussion section.

#### Notation:

Unless otherwise indicated, we use uppercase letters (e.g., $S_{t}$) for
random variables, lowercase letters (e.g., $s_{t}$) for values of random
variables, script letters (e.g., $\mathcal{S}$) for sets, and bold lowercase
letters (e.g., $\bm{\theta}$) for vectors. Let $[t_{1}:t_{2}]$ be the set of
integers $t$ such that $t_{1}\leq t\leq t_{2}$; we write $S_{t}$ such that
$t_{1}\leq t\leq t_{2}$ as $S_{t_{1}:t_{2}}$. We denote by
$\mathcal{N}(\mu,\sigma^{2})$ the normal distribution, where $\mu$ is the mean
and $\sigma$ the standard deviation. ${\mathbf{I}}$ denotes the identity
matrix. We denote the multivariate normal distribution with
$\mathcal{N}(\bm{\mu},\sigma^{2}{\mathbf{I}})$ where $\bm{\mu}$ is the mean
vector and the covariance matrix is diagonal with only $\sigma^{2}$ as
elements on the diagonal. Finally, $\mathbb{E}[\cdot]$ represents expectation
and $\mathbb{P}(\cdot)$ probability.

## Materials and Methods

(a)

(b)

(c)

Figure 1: Fig. 1(a) and 1(b) are two snapshots of the foraging game in which
the forager is about to collect coins. Fig. 1(c) shows a top view of the
environment where the coins are indicated by crosses and the coin clusters are
indicated by circles.

### Data Availability Statement

The dataset and the code used to generate our results are freely accessible at
our GitHub repository.

### Experimental Setup

In the following section, we provide a description of how the human foraging
datasets were collected. In the supplementary materials we include the full
dataset of $50$ search trajectories. We focus on five participants in the
context of a larger study investigating human foraging.[42] An example of two
foraging search trajectories is given in Fig. 2. All the experiments have been
carried out in accordance with the relevant guidelines and regulations and
approved by the Boston University’s Institutional Review Board.

#### Participants:

Participants consisted of male and female, neurologically healthy, English-
speaking volunteers between the ages of 18-35 with normal or corrected-to-
normal vision. Participants were recruited from Boston University and the
surrounding community. Individuals with a history of drug abuse, use of
psychoactive medication, neurological or psychiatric disorders, or learning
disabilities were excluded. Additionally, participants with a history of
motion sickness when watching or playing video games were also excluded. All
participants were compensated and gave written informed consent in accordance
with Boston University’s Institutional Review Board.

#### Task:

The task consisted of a $160m\times 160m$ virtual “open-field”, i.e. obstacle
free, paradigm surrounded by four differently colored and textured walls
created using Vizard 6.0, a Python-based virtual reality development platform
(Fig. 1). 325 coins were distributed throughout the environment, of which 100
were randomly distributed and 225 were distributed according to four different
multivariate Gaussian distributions of varying sizes: $75$ according to
$\mathcal{N}((60,75),5^{2}{\mathbf{I}})$, $40$ according to
$\mathcal{N}((-15,-50),11^{2}{\mathbf{I}})$, $60$ according to
$\mathcal{N}((-50,30),18^{2}{\mathbf{I}})$ and $50$ according to
$\mathcal{N}((49,-40),13^{2}{\mathbf{I}})$ (Fig. 1(c)). Each participant’s
starting location was randomized at the beginning of each run. Participants
could move forward and turn left or right. They could not move backwards. They
were instructed to freely explore the environment and collect as many coins as
possible but were not told anything about the distribution or total number of
coins. They were also able to see a running count of the coins they had
collected for each run. After being collected each coin disappears for the
remaining duration of the run. Participants performed the foraging task over
two consecutive days. On the first day, naive participants were presented with
the task on a desktop computer in a behavioral testing room. On the second
day, they performed the same task in an MRI scanner. Subjects performed 10
eight-minute runs on Day 1 and 10 eight-minute runs on Day 2. In the desktop
condition (Day 1), participants moved using keyboard arrow keys, and in the
scanner (Day 2), they moved using a diamond-shaped button box. For our
purposes here, we utilize the 80 minutes of behavioral data from Day 2 of the
experiment for 5 participants which collected an average of $243.98$ coins
each.

### Modeling the Human Decision Process

Figure 2: Two samples trajectories collected during the second day of tests.
The bar on the right shows the time in seconds. The full set of the $50$
trajectories is available in the supplementary materials.

In this section we describe the human modeling step and how the data are
processed to make them suitable for IL and RL.

We consider an infinite-horizon discounted Markov Decision Process (MDP)
defined by the tuple $(\mathcal{S},\mathcal{A},P,r,D,\gamma)$ where
$\mathcal{S}$ is the finite set of states and $\mathcal{A}$ is the finite set
of actions. $P:\mathcal{S}\times\mathcal{A}\rightarrow\Delta_{\mathcal{S}}$ is
the transition probability function and $\Delta_{\mathcal{S}}$ denotes the
space of probability distributions over $\mathcal{S}$. The function
$r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ maps rewards to state-
action pairs. $D\in\Delta_{\mathcal{S}}$ is the initial state distribution and
$\gamma\in[0,1)$ the discount factor. The decision agent is modeled as a
stationary policy $\pi:\mathcal{S}\rightarrow\Delta_{\mathcal{A}}$, where
$\pi(a|s)$ is the probability of taking action $a$ in state $s$. When a
deterministic policy is required we simply take
$a=\arg\max_{a^{\prime}}\pi(a^{\prime}|s)$. For simplicity, we will always
write $a\sim\pi(\cdot|s)$ and according to which algorithm we are referring to
it will be clear whether $\pi$ is stochastic or deterministic. We parameterize
$\pi$ using a neural network with parameters
$\bm{\theta}\in\varTheta\subset\mathbb{R}^{k}$ and we write
$\pi_{\bm{\theta}}$.

Given an MDP, we consider the human participants taking into account both
egocentric and allocentric strategies when navigating. [15, 43] We define the
state vector as $\bm{s}=\\{x,y,\psi,\chi\\}$, where $x,y$ are coordinates with
respect to a frame fixed to the environment and represent the allocentric
capacities of the agent, i.e., the ability to approximate its current position
within the environment. Instead, $\psi$ and $\chi$ are two categorical
variables that describe the human egocentric behavior: the first tells the
agent whether it can see a coin or not in its vicinity, $\psi\in\\{$ see coin,
no coins $\\}$, the second describes the "greedy" direction, i.e., the
direction of the closest coin the agent has in its view,
$\chi\in\\{\text{east},$ northeast, north, northwest, west, southwest, south,
southeast, no coins$\\}$.

The artificial agent perceives the state and takes an action to interact with
the environment. For computational reasons we discretize the $x,y$ coordinates
on a fine grid of $1m\times 1m$ and define the action space as
$a\in\\{\text{east},$ northeast, north, northwest, west, southwest, south,
$\text{southeast}\\}$. The transition to the next state always occurs
deterministically in the direction of the action $a$ taken by the agent. As
convention in Fig. 2, north means going from bottom to top, south from top to
bottom, east is left to right and west vice versa. The categorical state
$\psi$ stays $0$ all the time unless there is a coin in a radius of $8m$
distance, when $\psi$ turns $1$ then also $\chi$ turns from "no coins" to one
of the other directions. This is aligned with the original experiment where
each coin pops-up when the human is at $8m$ from it. Finally, the rewards are
simply represented by the coins in the environment where $r(\bm{s},a)=1$ for
each coin collected. As in the original experiment, the agents automatically
collect the reward once at $3m$ and $D$ is a uniform distribution over
$\mathcal{S}$.

### Imitation Learning

Given a task and an agent performing the task, IL infers the underlying agent
distribution via a set of an agent’s demonstrations (state-action samples).
Assuming the agent’s behavior is parameterized by a NN with optimal parameters
$\bm{\theta}^{*}$, we refer to the process of estimating $\bm{\theta}^{*}$
through a finite sequence of agent’s demonstrations
$\tau=(\bm{s}_{0:T},a_{0:T})$ with $2\leq T<\infty$ as IL. One way to
formulate this problem is through maximum likelihood estimation:

$\max_{\bm{\theta}}\mathcal{L}(\bm{\theta}),$ (1)

where $\mathcal{L}(\bm{\theta})$ denotes the log-likelihood and is equivalent
to the logarithm of the joint probability of generating the expert
demonstrations
$\tau=\\{\bm{s}_{0},a_{0},\bm{s}_{1},a_{1},\dots,\bm{s}_{T},a_{T}\\}$, i.e.,

$\mathcal{L}(\bm{\theta})=\log\mathbb{P}^{\bm{\theta}}_{D}(\tau).$ (2)

$\mathbb{P}^{\bm{\theta}}_{D}(\tau)$ in (2) is defined as

$\displaystyle\begin{split}&\mathbb{P}^{\bm{\theta}}_{D}(\tau)=D(\bm{s}_{0})\bigg{[}\prod_{t=0}^{T}\pi_{\bm{\theta}}(a_{t}|\bm{s}_{t})\bigg{]}\bigg{[}\prod_{t=0}^{T-1}P(\bm{s}_{t+1}|\bm{s}_{t},a_{t})\bigg{]}.\end{split}$
(3)

Computing the logarithm of (3) and neglecting the elements not parameterized
by $\bm{\theta}$ we obtain the following maximization problem

$\max_{\bm{\theta}}\sum_{t=0}^{T}\log\big{(}\pi_{\bm{\theta}}(a_{t}|\bm{s}_{t})\big{)}.$
(4)

Solving the maximization problem in Eq. (4) is the main objective of our IL
step.

### Reinforcement Learning

After defining a model, collecting the data and performing the imitation step,
our final goal is to further refine the imitated policies using RL. In RL, the
artificial agents are allowed to experience the task themselves and receive a
reinforcement according to the reward function $r(\bm{s}_{t},a_{t})$.
Mathematically, the goal is to find the policy parameters $\bm{\theta}$ which
maximize the expected total discounted reward
$J(\bm{\theta})=\mathbb{E}_{\tau}[\sum_{t=0}^{\infty}\gamma^{t}r(\bm{s}_{t},a_{t})]$,
where, as previously, $\tau=(\bm{s}_{0},a_{0},\bm{s}_{1},a_{1},\dots)$ is
sampled according to $\bm{s}_{0}\sim D$,
$a_{t}\sim\pi_{\bm{\theta}}(\cdot|\bm{s}_{t})$ and $\bm{s}_{t+1}\sim
P(\cdot|\bm{s}_{t},a_{t})$. Our focus is on model-free RL methods in which the
artificial agent does not know the transition probability function
$P(\cdot|\bm{s}_{t},a_{t})$, and it can only explore the environment and
experience rewards. Among these types of algorithms, we can distinguish two
main groups: $(i)$ algorithms that update the current policy following the
agent’s generated trajectories according to this policy, also known as on-
policy algorithms, and $(ii)$ algorithms that update the current policy using
experience from multiple policies used previously, known as off-policy. We
provide a more thorough introduction on this difference in the supplementary
materials.

State-of-art on-policy algorithms include Trust Region Policy Optimization
(TRPO),[44] and some robust variants such as Uncertainty Aware TRPO
(UATRPO),[45], and Proximal Policy Optimization (PPO).[44] Whereas, off-policy
methods include the Soft-Actor Critic (SAC) and Twin Delayed Deep
Deterministic Policy Gradient (TD3).[46, 47] All the mentioned algorithms are
tested in our experiments and more details about the NNs design are available
in the supplementary materials.

## Results

In this section, we present our results and describe all the steps that lead
to our design. All the code and data to replicate the experiments are freely
accessible at our GitHub repository
(https://github.com/VittorioGiammarino/Learning-from-humans-combining-
imitation-and-deep-on-policy-reinforcement-learning-to-accomplish-su). An
overview of the NNs used to parameterize $\pi_{\bm{\theta}}$ and all the
hyperparameters used for each of the IL and RL algorithms are in the
supplementary materials.

### Pre-processing

Our first step is to collect and process the $50$ trajectories, of $8$ minutes
each, recorded on the second day of tests. Each $8$-minute trajectory consists
of $28973$ data points, on average, which means we collect a data point every
$0.017s$, where the data points are the human agent’s coordinates with respect
to the fixed environment frame. Note that it is possible that a human agent
does not move for a few seconds, for example, and then makes many rapid
decisions about where to explore in the next milliseconds following this
stationary period. Therefore, the first "few seconds" could be aggregated in a
single data point while the next "milliseconds" would require more than a
single point. As a result, we aggregate the data points considering the
discretization of the $(x,y)$ coordinates. After that, we go over each of the
trajectories and determine the human decisions (i.e., for each aggregated
state, the direction of the human’s next movement). We cast each human
decision for each trajectory in the pre-determined action space $\mathcal{A}$
and construct in this way our state-action pairs (i.e., actions $a$ taken at
state $\bm{s}$). This process allows us to reduce the average length of human
trajectories from $28973$ to $3464$ data points without losing key
information. Note that this processing is an expensive but necessary step for
reducing the computational burden and enabling learning. Future research will
focus on how to automate this step and developing methods which can handle
learning from raw data.

### Imitation Learning

We perform IL on each human trajectory individually rather than considering a
single data set with all the trajectories. This is due to two main factors:
first, as Fig. 2 shows, each human trajectory covers the majority of the
environment; hence, each trajectory is per se informative enough about the
task. Second, the trajectories are noisy and a single aggregated data set
turns to be too noisy to be beneficial. The results of the IL step and all the
details on the evaluation are illustrated, for $5$ humans’ trajectories, in
Fig 3. In summary, we achieve good learning performance for several
trajectories but not enough to match the human participants. A figure showing
the IL performance for all the $50$ trajectories is available in the
supplementary materials.

Figure 3: The results from the IL step for $5$ different human trajectories
are illustrated. For each human trajectory we solve the IL problem for $8$
different seeds. For each seed, after every epoch we evaluate the performance
of the learnt policy for $10$ trials, each consisting of $3464$ steps. The
reported results for the "Imitation Learning Agent" show the reward averaged
over $10$ trials and $8$ seeds; the shaded area shows the standard deviation
over the seeds. The performance of the human trajectory used as data set for
the imitation is labelled as "Imitated Human". "Humans average" is the average
performance of the $50$ human trajectories.

### Reinforcement Learning

We consider the $50$ policies learnt from the $50$ human trajectories during
the IL step. We refine these policies using RL. We design the experiment as
follows:

  1. 1.

First, in order to determine which RL algorithm is more suitable for our goal,
we take the same single policy learnt during the IL step and use it as
initialization of each of the RL algorithms.

  2. 2.

Given the state-space dimension, we consider $10^{7}$ steps as a reasonable
amount of steps for performing RL.

  3. 3.

As in the IL step, for each RL algorithm we run the learning process for $8$
random seeds.

  4. 4.

During the learning process, we evaluate the policy learnt every $30,000$
steps on $10$ trials of $3464$ steps each. We report averaged results over the
$10$ trials and $8$ seeds. The shaded area in Fig. 4 and 5 shows the standard
deviation over seeds.

  5. 5.

After determining the most suitable RL algorithm, we rerun the whole
experiment for $50$ times in which each of the $50$ policies learnt during the
IL step is used as initialization of the selected RL algorithm.

Fig. 4 compares the various RL algorithms. PPO outperforms all other methods.
Broadly speaking, on-policy algorithms, i.e., PPO, TRPO, and UATRPO, learn
more effectively from a pre-initialized policy with respect to the off-policy
algorithms TD3 and SAC. Refer to the supplementary materials for more details.

Figure 4: The results for the RL algorithms initialized with the same policy
are illustrated. For each algorithm we run the learning process for $10^{7}$
steps and $8$ different random seeds. For each seed, after every $30,000$
steps we evaluate the performance of the learnt policy for $10$ trials each of
$3464$ steps. The reported results show the reward averaged over $10$ trials
and $8$ seeds, the shaded area shows the standard deviation over seeds.

Consequently, we proceed by combining IL together with PPO and compare it with
the PPO-only alternative. Fig. 5 illustrates the final results for the same
trajectories of Fig. 3 and a figure showing this final result for all the $50$
trajectories is available in the supplementary materials. In summary, IL
followed by PPO (IL+PPO) outperforms the average human performance and its
imitated expert, $39$ ($78\%$) and $31$ ($62\%$) times, respectively, over the
$50$ human trajectories. On the other hand, the PPO-only alternative cannot
get close to these results in $10^{7}$ steps. Table 1 summarizes the
comparison between humans and IL+PPO policies with respect to the total amount
of collectable rewards.

Note that, Fig. 5 provides interesting insights on why pre-training with IL
makes sense in foraging tasks with sparse rewards. We observe that, in
addition to a different initial performance, the IL+PPO and the PPO-only
agents show really different exploration strategies which lead to a different
reward convergence rate (the difference in rates is clearly visible in Fig.
5). The human-inspired exploration strategy of IL+PPO represents the main
strength of the method in this setup and the main source of difference with
the PPO-only agents.

Figure 5: The results of the IL+PPO method compared with the PPO-only alternative, the average human performance and the performance of the imitated expert are illustrated for the trajectories of Fig. 3. The experiment design and the reported results follow the same criterion as in Fig. 4. Table 1: A summary of the comparison between IL+PPO and human performance in the original experiment. The table shows the percentage out of $50$ learnt policies, for both the IL+PPO and the human agents, where the agent collected at least a certain percentage of rewards. As an example, the table is showing that, for the IL+PPO method, $19$ policies ($38\%$) can collect at least $260$ coins ($>80\%$). We compare these results with the human performance where each trajectory is considered as a single human policy. Note that, the IL+PPO policies perform better than humans on average but cannot do better than the best participants. |  | Performance Lower Bound
---|---|---
|  | $>70\%$ | $>80\%$ | $>85\%$ | $>90\%$
IL+PPO | $88\%$ | $38\%$ | $0\%$ | $0\%$
Humans | $80\%$ | $22\%$ | $10\%$ | $0\%$

### Robustness to reward distribution shift and the importance of egocentric
representations

(a) New rewards distribution.

(b) Old rewards distribution.

Figure 6: Fig. 6(a) shows the new reward distribution that was not previously seen by any of the artificial agents. We include the previous reward distribution in Fig. 6(b) to facilitate the comparison. Table 2: A summary of the results for the rewards distribution shift experiment. The table shows the fraction out of $50$ policies, for each initialization method, where after learning for $2\times 10^{6}$ steps, the agent is able to collect at least a certain percentage of rewards. As an example, the table is showing that, for the IL+PPO initialization, $46$ policies ($92\%$) can collect at least $270$ coins ($>80\%$) in this new scenario after learning for only $2\times 10^{6}$ steps.  |  | Performance Lower Bound
---|---|---
|  | $>70\%$ | $>80\%$ | $>90\%$ | $>95\%$
IL-only initialization | $28\%$ | $18\%$ | $0\%$ | $0\%$
IL+PPO initialization | $98\%$ | $92\%$ | $18\%$ | $0\%$

In this section, we test the learnt policies for robustness to a reward
distribution shift. The motivation is to explore how quickly the artificial
agents grasp changes in the environment and adapt to these changes. As in the
original experiment, $325$ coins are placed across the environment; however,
this time according to the new distribution in Fig. 6(a), we include the
original coins distribution in Fig. 6(b) to facilitate the comparison with
Fig. 6(a). Specifically, $50$ coins are distributed according to
$\mathcal{N}((-70,30),5^{2}{\mathbf{I}})$, $75$ according to
$\mathcal{N}((60,-20),11^{2}{\mathbf{I}})$, $100$ according to
$\mathcal{N}((-40,45),15^{2}{\mathbf{I}})$ and $100$ according to
$\mathcal{N}((0,60),13^{2}{\mathbf{I}})$.

We design the experiment similarly to the RL study and the IL+PPO experiments
in Fig. 4 and Fig. 5. Overall, we run, for $8$ different random seeds, $100$
learning experiments of $2\times 10^{6}$ steps each, where in the first $50$
we initialize using the policies learnt with only IL (Fig. 3), while, in the
second $50$, we initialize using the policies learnt by IL+PPO (Fig. 5). The
results are summarized in Table 2 and show that the policies learnt using both
IL+PPO generalize well to novel reward distributions. The figures showing the
detailed $100$ experiments are available in the supplementary materials.

In order to produce these results, we conclude that, given the state vector
representation as $\bm{s}=\\{x,y,\psi,\chi\\}$, the RL agents and their
exploration strategies must heavily rely on egocentric information, i.e., the
variables $\psi$ and $\chi$. This would explain the algorithm performance in
the novel reward environment in Fig. 6(a), where the previously learnt
allocentric representation is no longer informative. On the other hand, a
strategy based on egocentric exploration which facilitates the generation of a
new allocentric representation of the environment would explain the results in
Table 2. In other words, we suggest that our IL+PPO algorithms exhibit coding
of behavioral variables analogous to the observation in animals,[15] where
electrophysiological recording during foraging strategies indicate neural
coding in both egocentric and allocentric coordinate frames.

To demonstrate the veracity of this claim we rerun the entire set of
experiments, which includes IL as in Fig. 3 and IL+PPO as in Fig. 5, but this
time only providing allocentric information to the artificial agents. In other
words, we reduce the state vector from $\bm{s}=\\{x,y,\psi,\chi\\}$ to
$\bm{s}=\\{x,y\\}$. The results for a selected number of human trajectories
are summarized in Fig 7 and for the entire set of trajectories in the
supplementary materials. The final results show that agents trained with the
full state $\bm{s}=\\{x,y,\psi,\chi\\}$ outperform agents trained with the
allocentric only state $\bm{s}=\\{x,y\\}$ $74\%$ of the times for IL ($37$ out
of $50$) and $100\%$ of the times for IL+PPO. We conclude that our learnt
policies heavily rely on egocentric data and that the absence of such
information compromises to a large extent the learning performance as
illustrated in Fig 7.

Figure 7: Performance comparison, for selected trajectories, among agents
trained with full state including both egocentric and allocentric information
$\bm{s}=\\{x,y,\psi,\chi\\}$ versus an allocentric only state
$\bm{s}=\\{x,y\\}$. The comparison is done for both IL (upper figure) and
IL+PPO (lower figure). For completeness, we also show the PPO agent without IL
initialization performance (PPO allocentric and egocentric Agent).

## Discussion

In this paper, $50$ human navigation trajectories were collected in a virtual
open-field environment. We extracted a navigation control policy from each of
these trajectories and introduced an MDP setting to capture the navigational
human decision making. We learned policies consistent with the experimental
data using imitation learning based on log-likelihood maximization for each of
the trajectories.

After obtaining a control policy for each trajectory, we used all of them as a
starting point for RL, seeking to find policies that can efficiently
outperform the human participants in the same experimental setting. We tested
state-of-art on-policy (PPO, TRPO, UATRPO) and off-policy (TD3, SAC)
algorithms. We explained more extensively how these two categories differ in
the supplementary materials. Briefly, the main element of difference lies in
the data used to update the policy network $\pi_{\bm{\theta}}$ and in how we
compute and approximate the critic network. Off-policy algorithms are usually
faster to converge but introduce a large bias in the critic estimate, which
results in more oscillatory learning which often jeopardizes the IL
initialization. On the other hand, the tested on-policy algorithms are more
conservative, and the optimization step is constrained so not to diverge too
much from the current policy $\pi_{\bm{\theta}}$. This results in a slower but
more steady improvement of performance. Our preference towards PPO with
respect to the other on-policy algorithms is the result of empirical
experiments which corroborates other well-known empirical studies on the
matter.[48, 49]

Finally, we examined the sensitivity of the IL+PPO and IL-only policies to a
different reward distribution and investigated to what extent our artificial
agents rely on egocentric information. The final results showed that learning
only from data is not enough to match human performance and does not lead to
robustness over the reward distribution (Fig. 3 and Table 1). On the other
hand, IL followed by PPO (IL+PPO) showed impressive results in the original
experiment and it led to good generalization of the task (Fig. 5 and Table 2).
Further, we showed that such results are associated with the use of egocentric
information, which are crucial in enhancing learning performance both in the
IL and the IL+RL setting when compared with the use of allocentric information
alone.

In summary, we have developed a method to learn bio-inspired policies from
human navigation data, which can be further refined to achieve human-level
performance. This approach to modeling human navigational policies can be of
great utility for aerial and ground unmanned navigation tasks including
scientific exploration and search and rescue operations.

## Acknowledgments

Research was partially supported by the NSF under grants IIS-1914792,
DMS-1664644, CNS-1645681 and NSF-1829398, by the ONR under grants
N00014-19-1-2571 and N00014-21-1-2844, by the ONR DURIP under grants
N00014-17-1-2304, by the NIH under grants R01 GM135930 and UL54 TR004130, and
by the Boston University Kilachand Fund for Integrated Life Science and
Engineering.

## References

  * [1] Walker, C. M., Williams, J. J., Lombrozo, T. & Gopnik, A.  Explaining influences children’s reliance on evidence and prior knowledge in causal induction.  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , vol. 34 (2012).
  * [2] Gopnik, A., Griffiths, T. L. & Lucas, C. G.  When younger learners can be better (or at least more open-minded) than older ones.  _Current Directions in Psychological Science_ 24, 87–92 (2015).
  * [3] Goddu, M. K., Lombrozo, T. & Gopnik, A.  Transformations and transfer: Preschool children understand abstract relations and reason analogically in a causal task.  _Child Development_ 91, 1898–1915 (2020).
  * [4] Ruggeri, A., Pelz, M., Gopnik, A. & Schulz, E.  Toddlers search longer when there is more information to be gained.  _PsyArXiv preprint_ (2021).
  * [5] Silver, D. _et al._ Mastering the game of go without human knowledge.  _Nature_ 550, 354–359 (2017).
  * [6] Botvinick, M. _et al._ Reinforcement learning, fast and slow.  _Trends in Cognitive Sciences_ 23, 408–422 (2019).
  * [7] Offerman, T. & Sonnemans, J.  Learning by experience and learning by imitating successful others.  _Journal of Economic Behavior & Organization_ 34, 559–575 (1998).
  * [8] Jones, S. S.  The development of imitation in infancy.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ 364, 2325–2335 (2009).
  * [9] Pomerleau, D. A.  Efficient training of artificial neural networks for autonomous navigation.  _Neural Computation_ 3, 88–97 (1991).
  * [10] Abbeel, P., Coates, A. & Ng, A. Y.  Autonomous helicopter aerobatics through apprenticeship learning.  _The International Journal of Robotics Research_ 29, 1608–1639 (2010).
  * [11] Uchibe, E. & Doya, K.  Forward and inverse reinforcement learning sharing network weights and hyperparameters.  _Neural Networks_ 144, 138–153 (2021).
  * [12] Scone, S. & Phillips, I.  Trade-off between exploration and reporting victim locations in usar.  In _2010 IEEE International Symposium on" A World of Wireless, Mobile and Multimedia Networks"(WoWMoM)_ , 1–6 (IEEE, 2010).
  * [13] Otte, M., Correll, N. & Frazzoli, E.  Navigation with foraging.  In _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 3150–3157 (IEEE, 2013).
  * [14] cCatal, O., Verbelen, T., Van de Maele, T., Dhoedt, B. & Safron, A.  Robot navigation as hierarchical active inference.  _Neural Networks_ 142, 192–204 (2021).
  * [15] Alexander, A. S. _et al._ Egocentric boundary vector tuning of the retrosplenial cortex.  _Science Advances_ 6, eaaz2322 (2020).
  * [16] Schaal, S.  Is imitation learning the route to humanoid robots?  _Trends in Cognitive Sciences_ 3, 233–242 (1999).
  * [17] Syed, U. & Schapire, R. E.  A reduction from apprenticeship learning to classification.  _Advances in Neural Information Processing Systems_ 23 (2010).
  * [18] Ross, S. & Bagnell, D.  Efficient reductions for imitation learning.  In _Proceedings of the thirteenth International Conference on Artificial Intelligence and Statistics_ , 661–668 (JMLR Workshop and Conference Proceedings, 2010).
  * [19] Osa, T. _et al._ An algorithmic perspective on imitation learning.  _Foundations and Trends in Robotics_ 7, 1–179 (2018).
  * [20] Sutton, R. S. & Barto, A. G.  _Reinforcement learning: An introduction_ (MIT press, 2018).
  * [21] Dulac-Arnold, G., Mankowitz, D. & Hester, T.  Challenges of real-world reinforcement learning.  _arXiv preprint arXiv:1904.12901_ (2019).
  * [22] Ross, S., Gordon, G. & Bagnell, D.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _Proceedings of the fourteenth International Conference on Artificial Intelligence and Statistics_ , 627–635 (JMLR Workshop and Conference Proceedings, 2011).
  * [23] Ross, S. & Bagnell, J. A.  Reinforcement and imitation learning via interactive no-regret learning.  _arXiv preprint arXiv:1406.5979_ (2014).
  * [24] Sun, W., Bagnell, J. A. & Boots, B.  Truncated horizon policy search: combining reinforcement learning and imitation learning.  In _International Conference on Learning Representations_ (2018).
  * [25] Cheng, C., Yan, X., Wagener, N. & Boots, B.  Fast policy learning through imitation and reinforcement.  In _Uncertainty in Artificial Intelligence_ (2019).
  * [26] Subramanian, K., Isbell Jr, C. L. & Thomaz, A. L.  Exploration from demonstration for interactive reinforcement learning.  In _Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems_, 447–456 (2016).
  * [27] Vecerik, M. _et al._ Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.  _arXiv preprint arXiv:1707.08817_ (2017).
  * [28] Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W. & Abbeel, P.  Overcoming exploration in reinforcement learning with demonstrations.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 6292–6299 (IEEE, 2018).
  * [29] Libardi, G., De Fabritiis, G. & Dittert, S.  Guided exploration with proximal policy optimization using a single demonstration.  In _International Conference on Machine Learning_ , 6611–6620 (PMLR, 2021).
  * [30] Uchendu, I. _et al._ Demonstration-guided q-learning.  _NIPS Workshop on Robot Learning: Self-Supervised and Lifelong Learning_ (2021).
  * [31] Abbeel, P. & Ng, A. Y.  Apprenticeship learning via inverse reinforcement learning.  In _Proceedings of the twenty-first International Conference on Machine Learning_ , 1 (2004).
  * [32] Ratliff, N. D., Bagnell, J. A. & Zinkevich, M. A.  Maximum margin planning.  In _Proceedings of the twenty-third International Conference on Machine learning_ , 729–736 (2006).
  * [33] Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K. _et al._ Maximum entropy inverse reinforcement learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 8, 1433–1438 (Chicago, IL, USA, 2008).
  * [34] Finn, C., Levine, S. & Abbeel, P.  Guided cost learning: Deep inverse optimal control via policy optimization.  In _International Conference on Machine Learning_ , 49–58 (PMLR, 2016).
  * [35] Ho, J. & Ermon, S.  Generative adversarial imitation learning.  _Advances in Neural Information Processing Systems_ 29 (2016).
  * [36] Ghasemipour, S. K. S., Zemel, R. & Gu, S.  A divergence minimization perspective on imitation learning methods.  In _Conference on Robot Learning_ , 1259–1277 (PMLR, 2020).
  * [37] Goodfellow, I. _et al._ Generative adversarial networks.  _Communications of the ACM_ 63, 139–144 (2020).
  * [38] Kang, B., Jie, Z. & Feng, J.  Policy optimization with demonstrations.  In _International Conference on Machine Learning_ , 2469–2478 (PMLR, 2018).
  * [39] Hester, T. _et al._ Deep q-learning from demonstrations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32 (2018).
  * [40] Mnih, V. _et al._ Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ (2013).
  * [41] Levine, S., Kumar, A., Tucker, G. & Fu, J.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ (2020).
  * [42] Moore, K., Yi, C., Dunne, M., Stern, C. & McGuire, J.  Virtual human foraging behavior follows predictions for heavy-tailed search.  _In Society for Neuroscience_ Online (2021).
  * [43] Feigenbaum, J. D. & Morris, R. G.  Allocentric versus egocentric spatial memory after unilateral temporal lobectomy in humans.  _Neuropsychology_ 18, 462 (2004).
  * [44] Schulman, J., Levine, S., Abbeel, P., Jordan, M. & Moritz, P.  Trust region policy optimization.  In _International Conference on Machine Learning_ , 1889–1897 (PMLR, 2015).
  * [45] Queeney, J., Paschalidis, I. C. & Cassandras, C. G.  Uncertainty-aware policy optimization: A robust, adaptive trust region approach.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, 9377–9385 (2021).
  * [46] Haarnoja, T., Zhou, A., Abbeel, P. & Levine, S.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International Conference on Machine Learning_ , 1861–1870 (PMLR, 2018).
  * [47] Fujimoto, S., Hoof, H. & Meger, D.  Addressing function approximation error in actor-critic methods.  In _International Conference on Machine Learning_ , 1587–1596 (PMLR, 2018).
  * [48] Andrychowicz, M. _et al._ What matters in on-policy reinforcement learning? a large-scale empirical study.  _arXiv preprint arXiv:2006.05990_ (2020).
  * [49] Engstrom, L. _et al._ Implementation matters in deep policy gradients: A case study on ppo and trpo.  In _International Conference on Learning Representations_ (2020).
  * [50] Sutton, R. S., McAllester, D. A., Singh, S. P., Mansour, Y. _et al._ Policy gradient methods for reinforcement learning with function approximation.  In _NIPs_ , vol. 99, 1057–1063 (Citeseer, 1999).
  * [51] Queeney, J., Paschalidis, I. & Cassandras, C.  Generalized proximal policy optimization with sample reuse.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * [52] Kakade, S. & Langford, J.  Approximately optimal approximate reinforcement learning.  In _In Proc. 19th International Conference on Machine Learning_ (Citeseer, 2002).

## Appendix

### On-policy vs. off-policy RL algorithms

The Dichotomy between on-policy and off-policy reinforcement learning (RL)
algorithms started since the earliest days of RL and specifically with
temporal-difference (TD) learning.[20].

Recall that the goal of RL is to find $\bm{\theta}$ such that the expected
total discounted reward
$J(\bm{\theta})=\mathbb{E}_{\tau}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})]$
is maximized and where $\tau=(s_{0},a_{0},s_{1},a_{1},\dots)$ is sampled
according to $s_{0}\sim D$, $a_{t}\sim\pi_{\bm{\theta}}(\cdot|s_{t})$ and
$s_{t+1}\sim P(\cdot|s_{t},a_{t})$. The mechanism used in all the mentioned
algorithms consists of alternating policy evaluation with policy improvement
and doing it for several iterations until convergence. Specifically, the
policy evaluation step consists of using the agent’s policy
$\pi_{\bm{\theta}}$ to generate trajectories
$\tau=(s_{0},a_{0},s_{1},a_{1},\dots)$ to estimate the state-action value
function denoted by
$Q_{\bm{\theta}}(s,a)=\mathbb{E}_{\tau}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})|S_{0}=s,A_{0}=a]$,
and/or the state value function
$V_{\bm{\theta}}(s)=\mathbb{E}_{a\sim\pi_{\bm{\theta}}(\cdot|s_{t})}[Q_{\bm{\theta}}(s,a)]$.
Then, the policy improvement step, updates the current policy
$\pi_{\bm{\theta}}$ via a policy gradient in a direction that maximizes
$J(\bm{\theta})$. [50]

On-policy algorithms rely only on trajectories $\tau$ generated by the
interaction of $\pi_{\bm{\theta}}$ with the environment at the current
iteration. Whereas, off-policy algorithms retrieve trajectories generated also
at previous iterations and use this past experience in the policy update. This
has indeed a different connotation in practice.[51] On-policy methods deliver
stable performance throughout learning due to their connection to theoretical
policy improvement guarantees.[52] However, theoretically supported stability
yields high sample complexity due to the conspicuous amount of trajectories
needed at each iteration. On the other hand, off-policy algorithms address the
sample complexity issue by storing data in a replay buffer which are then
reused for multiple policy updates. Off-policy methods have proved to be more
sample efficient in practice but also more unstable due to the bias introduced
in estimating the state-action value function $Q_{\bm{\theta}}(s,a)$. A recent
work has sought to combine on-policy algorithms with off-policy ideas.[51]

In the specific case of this paper, we observe that the instability due to
off-policy updates can undermine the imitation learning (IL) initialization as
illustrated in Fig. 8 (see SAC and TD3). At the same time, the IL
initialization remarkably helps on-policy methods to accomplish fast and
efficient convergence. Thus, IL+PPO results in a more reliable and efficient
learning method.

Figure 8: The results for the RL algorithms initialized with the same
imitation learning policy.

### Neural Network design

We use the same $\pi_{\bm{\theta}}$ architecture for all the algorithms. On-
policy algorithms rely on Generalized Advantage Estimation to compute their
critic, this method trade-offs between high variance Monte-Carlo estimation
and biased Bootstrap and relies only on the value function network
$V_{\bm{\theta}_{v}}$. On the other hand, off-policy algorithms use TD
learning which relies on a state-action value function $Q_{\bm{\theta}_{q}}$.
All the architectures are summarized in Table 3.

Table 3: Neural Networks architectures. $\pi_{\bm{\theta}}$ network
architecture
---
Layer type | Dimension | Activation Function
Input | observation size | Linear
Hidden fully connected | 128 | Relu
Output | action space cardinality | Softmax
$V_{\bm{\theta}_{v}}$ network architecture.
Layer type | Dimension | Activation Function
Input | observation size | Linear
Hidden fully connected | 256 | Relu
Output | 1 | Linear
$Q_{\bm{\theta}_{q}}$ network architecture.
Layer type | Dimension | Activation Function
Input | observation size | Linear
Hidden fully connected | 256 | Relu
Output | action space cardinality | Linear

### Hyperparameter values, human trajectories and experiments

Table 4: Hyperparameter values for experimental setup. Evaluation | Default
---|---
Episodes per evaluations | 10
Steps per evaluation episode | 3464
General IL |  |
Optimization algorithm | Minibatch gradient ascent
Size minibatches | 32
Epochs per update | 1
Number of updates | 11
Optimizer | Adam
Learning rate | 0.001
General RL on-policy |  |
Max number of steps | $10.02\times 10^{6}$
Steps between evaluations | $30000$
GAE $\gamma$ | 0.99
GAE $\lambda$ | 0.99
PPO |  |
Clipping parameter $\epsilon$ | 0.2
Entropy | True
Entropy weight $c_{2}$ | $10^{-2}$
Optimization algorithm | Minibatch gradient ascent
Batch size | $30000$
Size minibatches | 64
Epochs per update | 10
Optimizer | Adam
Learning rate | $3\times 10^{-4}$
TRPO |  |
Trust region parameter $\epsilon$ | 0.05
Entropy | False
Optimization algorithm | Conjugate gradient ascent with line search
Conjugate gradient damping | $0.1$
Conjugate iterations per update | $20$
UATRPO |  |
Trust region parameter $\epsilon$ | 0.03
Entropy | False
Optimization algorithm | Conjugate gradient ascent with line search
Conjugate gradient damping | $0.1$
Conjugate gradient iterations per update | $20$
Trade-off Parameter (c) | $6\times 10^{-4}$
Confidence parameter ($\alpha$) | $0.05$
Number of random projections ($m$) | $200$
EMA weight parameter ($\beta$) | $0.9$
Table 5: Hyperparameter values for experimental setup (Cont.). General RL off-policy | Default
---|---
Max number of steps | $10.02\times 10^{6}$
Steps between evaluations | $30000$
Buffer size | $10^{6}$
Discount $\gamma$ | 0.99
Optimization algorithm | Batch gradient ascent
Batch size | $256$
Optimizer | Adam
Learning rate | $3\times 10^{-4}$
SAC |  |
Entropy temperature init $\alpha$ | $0.2$
Target critic update step ($\tau$) | $0.005$
TD3 |  |
Exploration noise std | $0.1$
Policy Noise std | $0.2$
Noise clip | $0.5$
Update steps delay | $2$
Target critic and policy update step ($\tau$) | $0.005$
Figure 9: The complete set of $50$ 8-minutes human trajectories collected
during the $2^{\text{nd}}$ day of tests.

Figure 10: Results of the IL step for all the $50$ human trajectories in Fig.
9. Figure 11: Results of the PPO step with the $50$ IL policies in Fig. 10.
Figure 12: Results of learning a new rewards distribution in Fig 6(a) with the
IL-only initialization for all the $50$ IL policies in Fig. 10. Figure 13:
Results of learning a new rewards distribution in Fig 6(a) with IL+PPO
initialization for all the $50$ policies in Fig. 11. Figure 14: Comparison
between IL with full state and allocentric only state for all the $50$ human
trajectories. Figure 15: Comparison between IL+PPO with full state and
allocentric only state for all the $50$ human trajectories. Figure 16:
Comparison between IL with full state, allocentric only state and egocentric
only state for all the $50$ human trajectories. Figure 17: Comparison between
IL+PPO with full state, allocentric only state and egocentric only state for
all the $50$ human trajectories.