# A learning gap between neuroscience
and reinforcement learning

Samuel T. Wauthier, Pietro Mazzaglia, Ozan Çatal, Cedric De Boom, Tim Verbelen
&
Bart Dhoedt
IDLab, Department of Information Technology
Ghent University – imec
Technologiepark-Zwijnaarde 126, B-9052 Ghent, Belgium
{samuel.wauthier, pietro.mazzaglia, ozan.catal, cedric.deboom,
tim.verbelen<EMAIL_ADDRESS>
###### Abstract

Historically, artificial intelligence has drawn much inspiration from
neuroscience to fuel advances in the field. However, current progress in
reinforcement learning is largely focused on benchmark problems that fail to
capture many of the aspects that are of interest in neuroscience today. We
illustrate this point by extending a T-maze task from neuroscience for use
with reinforcement learning algorithms, and show that state-of-the-art
algorithms are not capable of solving this problem. Finally, we point out
where insights from neuroscience could help explain some of the issues
encountered.

##  1 Introduction

Neuroscience and artificial intelligence have a long-standing history of
cross-fertilization between the two domains (Hassabis et al., 2017). Most
notable are advances in deep reinforcement learning (RL), which is heavily
inspired by reward prediction error signals observed in the brain, as well as
mimicking neural circuits for computation (Mnih et al., 2015). Current RL
algorithms are even surpassing human performance on various, typically game
environments such as Atari games (Badia et al., 2020), Dota 2 (Berner et al.,
2019) or Go (Silver et al., 2016). However, these latest achievements are
mainly attributed to incremental updates to the RL algorithm and scaling up
training to huge amounts of data.

Progress in artificial intelligence is often driven by benchmark problems. A
clear example is the ImageNet data set, which is driving progress in computer
vision (Krizhevsky et al., 2017; He et al., 2016; Khan et al., 2020). Also in
RL, standard benchmarks were introduced to measure performance of various
algorithms, such as the Atari Learning Environment (Bellemare et al., 2013),
Open AI Gym (Brockman et al., 2016), the DeepMind Lab (Beattie et al., 2016)
and Control Suite (Tassa et al., 2018). Such benchmark problems become
increasingly harder to solve, with the main focus being increasing the
complexity of observations (i.e. from control states to pixels), and
increasing the difficulty of finding rewards (i.e. sparse reward signals).
However, we argue that many aspects that are of interest in (behavioral)
neuroscience, such as dealing with ambiguity, stochasticity, and memory, are
less pronounced.

We propose that, once again, we should draw more inspiration from neuroscience
and build novel benchmark problems that are, on the one hand, simple enough to
enable relatively fast iterations for algorithm development, but on the other
hand, hard enough for current algorithms to tackle. One notable example in
this direction is the Animal AI testbed (Beyret et al., 2019), which focuses
on spatial navigation and tasks involving inference of object locations
through object persistence and spatial elimination. We believe that grounding
RL benchmark problems in well-studied problems in neuroscience will also
foster novel algorithms inspired by neuroscience theories and, at the same
time, provide empirical evidence for these theories.

Figure 1: Foraging environment as defined by Friston et al. (2016). The left
and right images represent two possible contexts. The agent starts in the
middle. The red circle indicates the location of the reward. The green and
blue circles are clues that tell the agent which context it is in, e.g. blue
and green mean the reward will be on the left and right, respectively.

Psychology and neuroscience have been studying rodent behavior in mazes since
the early 20th century (Tolman & Honzik, 1930; O’Keefe & Dostrovsky, 1971). A
more recent example is the (artificial) T-maze, described by Friston et al.
(2016). In this environment, an artificial agent, e.g. a rat, is put inside a
T-shaped maze with a single reward, e.g. some cheese, in either the left or
the right branch (see Figure 1). The lower branch contains a clue on where the
reward is located. The agent starts in the center of the world and can choose
which branch to take. Crucially, after it has entered the left or right
branch, it is not allowed to exit this branch. The initial state of the world
is ambiguous, since the agent has no way of knowing its context, i.e. which
type of world it finds itself in: a world with a reward in the left branch or
a world with a reward in the right branch. In addition, without knowledge of
whether the right or the left branch contains the reward, the agent can only
guess which branch to take. Observing the clue in the lower branch allows the
agent to resolve the ambiguity and make an informed decision. This T-maze was
originally described in a discrete way, i.e. the agent can be in four possible
locations (center, bottom, left, right) and has a maximum of two moves. Note
that the agent is allowed to go directly to the left or right branch from the
bottom branch. In their work, Friston et al. (2016) highlight that the key to
solving the problem is the use of belief-based methods, whereas belief-free
methods, such as dynamic programming, fail.

In the remainder of this paper, we revisit the environment described by
Friston et al. (2016) and reimplement it as a pixel-based, game-like
environment similar to the Atari benchmark. Furthermore, we show that a set of
state-of-the-art reinforcement learning algorithms is unable to solve this
environment. We conclude by discussing some of the limitations of current RL
approaches and point out where inspiration from neuroscience might help.

##  2 E-maze: T-maze revisited

Figure 2: (left) Starting position in E-maze (with and without black areas).
(right) Path blocked after the agent enters a branch (with and without black
areas).

In this section, we extend the T-maze of Friston et al. (2016) in order to be
compatible with current RL benchmarks111Environment source code is available
at https://github.com/thesmartrobot/ambigym.. For example, we use pixel-based
observations and allow the agent to move to intermediate positions between
endpoints. In other words, observations become images, and the state space
becomes larger. As a direct result, the maximum number of moves grows as well,
since the environment cannot be solved in two moves anymore.

Our environment is shown in Figure 2. It consists of a top-down view of a
tilted E-shaped maze. The grey triangle marks the agent, while the yellow
circle indicates the reward. Walls are displayed in red, and windows are
displayed in blue. Areas that the agent is unable to see are filled with
black. The agent can see through windows, but not through walls. Figure 2 also
shows the environment with all the black-filled areas removed for the purpose
of demonstration. The agent’s field of view (FOV) and maximum number of moves
are hyperparameters. Possible actions are: do nothing, move forward, and turn
left or right by 45 degrees. The moment the agent enters the left or right
branch, the branch closes, and the agent cannot turn back (see Figure 2).

Note that the environment can be modeled as a partially observable Markov
decision process (POMDP). The agent is unable to infer the underlying state
directly from a single observation, since areas that are not within the line
of sight of the agent are invisible to the agent. Moreover, from an RL
standpoint, this setup corresponds to a sparse reward environment. Indeed, no
reward is gained unless the agent obtains the yellow circle. Only if the agent
reaches the yellow circle, it receives a reward of 1 and the episode is
stopped.

##  3 Experiments

A number of experiments were set up to verify the performance of different RL
algorithms on the environment. The environment was initialized in the same
manner for all experiments. It was evaluated on model-free approaches (DQN
(Mnih et al., 2015), Rainbow (Hessel et al., 2018), PPO (Schulman et al.,
2017)), PPO with exploration bonuses (ICM (Pathak et al., 2017), RND (Burda et
al., 2019b)) and a model-based approach (DreamerV2 (Hafner et al., 2021)).
Details can be found in Appendix A and B.

Performance was measured by how often the agent was able to reach the reward.
In practice, a rolling average of the returns over 100 episodes was used for
the evaluation, where return denotes the episodic sum of rewards. Since the
return for this environment is either 0 or 1, the rolling average estimates
the frequency with which the agent is able to reach the reward. Furthermore,
we average over 10 different training runs for each algorithm. In what
follows, we shall refer to ‘performance’ to indicate the average number of
times the agent was able to reach the reward.

Figure 3: Mean rolling average of episode return for different algorithms.
Multiple runs were done for each algorithm and a rolling average was taken
over 100 episodes for each run, after which the mean over all runs was taken.
Shaded areas show the range between the minimum and maximum rolling average.

Given the random nature of the environment, a random agent with an unbounded
number of allowed moves would reach the reward 50% of the time. In other
words, with the performance measure defined above, the algorithm under
consideration must reach the reward more than 50% of the time to be able to
say that it performs better than a random agent. This is important, since an
RL agent is not guaranteed 50% performance when the maximum number of moves is
small. For example, it may never learn to go towards the reward due to never
reaching the reward.

Figure 3 shows the mean rolling average of return per episode for the
algorithms that learned to go to the reward over the course of training.
Shaded areas indicate the range between the minimum and maximum rolling
average over all runs. DQN and Rainbow displayed the same behaviour and are
shown in the same plot.

Typical behavior of the rolling average curves of individual runs is that
either the agent fails to learn and stays at 0%, or the agent jumps to the
level of a random agent from 0% to around 50%. This is reflected in the minima
and maxima. For PPO and the exploration methods, whether the agent fails to
learn depends on the seed. As such, the mean over all runs reflects the amount
of seeds for which the agent failed to learn. For example, PPO reached a mean
rolling average of 10% over 10 runs, indicating that it failed 8 out of 10
times.

Out of the three model-free methods that were assessed, DQN and Rainbow fail
to learn, while PPO (without orthogonal initialization or exploration)
achieves very low performance. The model-free methods suffer most from the
sparse reward task and are unable to learn when the reward is not reached
through random actions. Adding exploration bonuses improves performance for
PPO by 20–30% on average. Additionally, we found that using orthogonal weight
initialization (Saxe et al., 2013) drastically improves performance on PPO.
This is due to the fact that this yields more diverse initial random behavior.
Finally, DreamerV2 achieved 50% after about 2000 episodes, but never increased
to more than 50%.

##  4 Discussion

The results show that model-free methods fail without specific initialization
or exploration method. On the one hand, this is a demonstration of how
sensitive RL methods are to hyperparameters and how clever “tricks” must be
used to obtain good results. State-of-the-art results are usually obtained
through important details, such as proper initialization, input normalization
or adaptive learning techniques (Rao et al., 2020). These details are often
not highlighted as being crucial for the performance of the algorithm. As a
result, one could argue whether there is a default setting for these
algorithms that generalizes well to any environment. The right tuning is
usually necessary to obtain good results.

On the other hand, methods that use a model, either for providing an
exploration bonus or model-based approaches, at least succeed at reaching the
reward and getting 50% performance, regardless of initialization. From a
neuroscience perspective, one could point to theories in which the brain
contains a model of the world, i.e. the brain maintains beliefs of the world
through a generative model, such as Bayesian approaches to brain function
(Erickson & Smith, 1988), active inference (Friston et al., 2006), and the
general adversarial brain (Gershman, 2019). These models allow us to react to
events with a stochastic nature and where information is not perfect. Perhaps
the use of a world model is crucial for algorithms to generalize well to any
environment.

One important aspect of the environment is that rewards are sparse. When
rewards are sparse and difficult to find through random exploration, model-
free approaches typically fail. In that case, some incentive is needed to
explore. This incentive can be given through reward-shaping, intrinsic reward
from exploration methods or, as mentioned earlier, clever initialization.
White (1959) argues that exploratory behavior in itself can be a source of
reward. For example, the theory of flow (Csikszentmihalyi, 1990) states that
an important source of intrinsic reward for humans is the interest in
activities which require slightly more skill than they currently possess.
Active inference (Friston et al., 2006) frames the minimization of
(variational) free energy as the intrinsic reward that governs the actions of
living things. The inclusion of an intrinsic reward, in addition to an
extrinsic reward, may be a necessity.

Another important aspect of the environment is the relatively long time frame
between seeing the reward through the window and obtaining the reward. To
optimally solve the E-maze, the agent will first need to go towards the
windows to find out where the reward is, and then, remember the location when
it chooses a branch. Clearly, the agent will need some kind of memory to
achieve this. In some sense, model-based methods contain a type of implicit
memory encoded in the state, though they can also use explicit memory, such as
long short-term memory (LSTM), gated recurrent units (GRUs), memory networks
(Weston et al., 2014) or neural Turing machines (Graves et al., 2014).
However, as shown in the previous section, the model-based method DreamerV2
did not perform better than an agent which randomly chooses a branch without
looking for the clue. This may suggest the need to look to types of memory
grounded in neuroscience, such as Hopfield networks (Little, 1974) or Kanerva
machines (Wu et al., 2018).

Reinforcement learning needs to look to neuroscience for inspiration once
again. Current benchmarks are too restrictive in terms of the types of
problems they address. We introduced the E-maze, which was unable to be solved
by state-of-the-art deep RL algorithms. Finally, we argued how insights from
neuroscience may aid in solving the E-maze. The use of world models, intrinsic
rewards, and memory, rooted in neuroscience, will be beneficial in this
regard.

### Acknowledgments

This research received funding from the Flemish Government under the
“Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.
Ozan Çatal is funded by a Ph.D grant of the Flanders Research Foundation
(FWO).

## References

  * Badia et al. (2020) Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell.  Agent57: Outperforming the atari human benchmark, 2020.
  * Beattie et al. (2016) Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen.  Deepmind lab.  _CoRR_ , abs/1612.03801, 2016.  URL http://arxiv.org/abs/1612.03801.
  * Bellemare et al. (2013) Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  _J. Artif. Int. Res._ , 47(1):253–279, May 2013\.  ISSN 1076-9757.
  * Bellemare et al. (2017) Marc G. Bellemare, Will Dabney, and Rémi Munos.  A distributional perspective on reinforcement learning.  In Doina Precup and Yee Whye Teh (eds.), _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 449–458, International Convention Centre, Sydney, Australia, 06–11 Aug 2017. PMLR.
  * Berner et al. (2019) Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang.  Dota 2 with large scale deep reinforcement learning.  _CoRR_ , abs/1912.06680, 2019.  URL http://arxiv.org/abs/1912.06680.
  * Beyret et al. (2019) Benjamin Beyret, José Hernández-Orallo, Lucy Cheke, Marta Halina, Murray Shanahan, and Matthew Crosby.  The animal-ai environment: Training and testing animal-like artificial cognition.  _CoRR_ , abs/1909.07483, 2019.  URL http://arxiv.org/abs/1909.07483.
  * Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.  _CoRR_ , abs/1606.01540, 2016.  URL http://arxiv.org/abs/1606.01540.
  * Burda et al. (2019a) Yuri Burda, Harrison Edwards, Deepak Pathak, Amos J. Storkey, Trevor Darrell, and Alexei A. Efros.  Large-scale study of curiosity-driven learning.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019a.
  * Burda et al. (2019b) Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.  Exploration by random network distillation.  In _International Conference on Learning Representations_ , 2019b.
  * Castro et al. (2018) Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare.  Dopamine: A research framework for deep reinforcement learning, 2018.
  * Csikszentmihalyi (1990) Mihaly Csikszentmihalyi.  _Flow: the Psychology of Optimal Experience by Mihaly Csikszentmihalyi_.  Harper & Row, 1990.  ISBN 9780060162535.
  * Dhariwal et al. (2017) Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov.  Openai baselines.  https://github.com/openai/baselines, 2017.
  * Erickson & Smith (1988) G. Erickson and C.R. Smith.  _Maximum-Entropy and Bayesian Methods in Science and Engineering: Foundations_.  Fundamental Theories of Physics. Springer Netherlands, 1988.  ISBN 9789027727930.
  * Friston et al. (2006) Karl Friston, James Kilner, and Lee Harrison.  A free energy principle for the brain.  _Journal of Physiology-Paris_ , 100(1):70–87, 2006.  ISSN 0928-4257.  doi: 10.1016/j.jphysparis.2006.10.001.  Theoretical and Computational Neuroscience: Understanding Brain Functions.
  * Friston et al. (2016) Karl Friston, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, John O’Doherty, and Giovanni Pezzulo.  Active inference and learning.  _Neuroscience & Biobehavioral Reviews_, 68:862–879, Sep 2016.  ISSN 01497634.  doi: 10.1016/j.neubiorev.2016.06.022.
  * Gershman (2019) Samuel J. Gershman.  The generative adversarial brain.  _Frontiers in Artificial Intelligence_ , 2:18, 2019.  ISSN 2624-8212.  doi: 10.3389/frai.2019.00018.
  * Graves et al. (2014) Alex Graves, Greg Wayne, and Ivo Danihelka.  Neural Turing Machines.  _arXiv e-prints_ , art. arXiv:1410.5401, October 2014.
  * Hafner et al. (2020) Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi.  Dream to control: Learning behaviors by latent imagination.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.
  * Hafner et al. (2021) Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba.  Mastering atari with discrete world models.  In _International Conference on Learning Representations_ , 2021.
  * Hassabis et al. (2017) Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick.  Neuroscience-inspired artificial intelligence.  _Neuron_ , 95(2):245–258, 2017.  ISSN 0896-6273.  doi: https://doi.org/10.1016/j.neuron.2017.06.011.  URL https://www.sciencedirect.com/science/article/pii/S0896627317305093.
  * He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 770–778, 2016.  doi: 10.1109/CVPR.2016.90.
  * Hessel et al. (2018) Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.  Rainbow: Combining improvements in deep reinforcement learning.  In _AAAI Conference on Artificial Intelligence_ , 2018.
  * Khan et al. (2020) Asifullah Khan, Anabia Sohail, Umme Zahoora, and Aqsa Saeed Qureshi.  A survey of the recent architectures of deep convolutional neural networks.  _Artificial Intelligence Review_ , 53(8):5455–5516, Dec 2020.  ISSN 1573-7462.  doi: 10.1007/s10462-020-09825-6.
  * Krizhevsky et al. (2017) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  _Commun. ACM_ , 60(6):84–90, May 2017.  ISSN 0001-0782.  doi: 10.1145/3065386.
  * Little (1974) W.A. Little.  The existence of persistent states in the brain.  _Mathematical Biosciences_ , 19(1):101–120, 1974\.  ISSN 0025-5564.  doi: 10.1016/0025-5564(74)90031-5.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, 2015.
  * Mnih et al. (2016) Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In Maria Florina Balcan and Kilian Q. Weinberger (eds.), _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pp. 1928–1937, New York, New York, USA, 20–22 Jun 2016. PMLR.
  * O’Keefe & Dostrovsky (1971) J. O’Keefe and J. Dostrovsky.  The hippocampus as a spatial map. preliminary evidence from unit activity in the freely-moving rat.  _Brain Research_ , 34(1):171–175, 1971.  ISSN 0006-8993.  doi: 10.1016/0006-8993(71)90358-1.
  * Pathak et al. (2017) Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell.  Curiosity-driven exploration by self-supervised prediction.  In Doina Precup and Yee Whye Teh (eds.), _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 2778–2787, International Convention Centre, Sydney, Australia, 06–11 Aug 2017. PMLR.
  * Rao et al. (2020) Nirnai Rao, Elie Aljalbout, Axel Sauer, and Sami Haddadin.  How to Make Deep RL Work in Practice.  _arXiv e-prints_ , art. arXiv:2010.13083, October 2020.
  * Saxe et al. (2013) Andrew M. Saxe, James L. McClelland, and Surya Ganguli.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.  _arXiv e-prints_ , art. arXiv:1312.6120, December 2013.
  * Schaul et al. (2016) Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay, 2016.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _CoRR_ , abs/1707.06347, 2017.
  * Silver et al. (2016) David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis.  Mastering the game of Go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, jan 2016.  ISSN 0028-0836.  doi: 10.1038/nature16961.
  * Sutton & Barto (2018) Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  2018\.
  * Tassa et al. (2018) Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller.  Deepmind control suite.  _CoRR_ , abs/1801.00690, 2018.  URL http://arxiv.org/abs/1801.00690.
  * Tolman & Honzik (1930) E. C. Tolman and C. H. Honzik.  Introduction and removal of reward, and maze performance in rats.  _University of California Publications in Psychology_ , 4:257–275, 1930.
  * Weston et al. (2014) Jason Weston, Sumit Chopra, and Antoine Bordes.  Memory Networks.  _arXiv e-prints_ , art. arXiv:1410.3916, October 2014.
  * White (1959) R. W. White.  Motivation reconsidered: the concept of competence.  _Psychological review_ , 66:297–333, 1959.
  * Wu et al. (2018) Yan Wu, Greg Wayne, Alex Graves, and Timothy Lillicrap.  The kanerva machine: A generative distributed memory.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=S1HlA-ZAZ.

##  Appendix A Algorithm descriptions

The following state-of-the-art RL algorithms were used to evaluate performance
in our environment.

DQN Deep Q Network (DQN; Mnih et al. (2015)) is a value-based optimization
method. The data collected from the environment is used to train a neural
network predicting the action-value function $Q$, which provides estimates of
the expected returns for all possible actions. The policy is implemented as a
selection process of the action with the highest $Q$ value at each time step.
For DQN, some exploration is provided through the $\epsilon$-greedy strategy,
which consists of choosing a random action rather than the best one with
probability $1-\epsilon$.

Rainbow Several improvements over the original DQN implementation have been
combined in (Hessel et al., 2018), leading to an overarching system referred
to as ‘Rainbow’. The three most important components of Rainbow are: the use
of n-step returns (Mnih et al., 2016), the use of a prioritized experience
replay (Schaul et al., 2016) and modeling the Q function as a distribution
(Bellemare et al., 2017).

Both our DQN and Rainbow implementations rely on the Dopamine RL framework
(Castro et al., 2018), which provides state-of-the-art results on the Atari
2600 benchmark.

PPO The Proximal Policy Optimization (PPO; Schulman et al. (2017)) algorithm
is a policy-gradient method for deep RL. Policy-gradient methods exploit the
Policy Gradient theorem (Sutton & Barto, 2018) to directly update the policy
based on the environment returns. With respect to Vanilla Policy Gradient (VPG
or REINFORCE), PPO implements two additional aspects: a value function
estimation, which is subtracted from the returns of the environment to reduce
variance, and a clipped objective function, which prevents undesirable drastic
updates of the policy.

Our implementation of the PPO algorithm was based upon the original OpenAI
Baselines implementation (Dhariwal et al., 2017).

Dreamer Dreamer (Hafner et al., 2020) is a model-based RL method based on two
main principles: (i) learning a ‘world’ model of the environment, which allows
predicting future observations and rewards, (ii) using the world model to
learn an optimal policy in an RL fashion, by applying policy-gradient updates
over imaginary trajectories generated by the Dreamer’s world model.

In our experiments, we ground on the second iteration of the Dreamer agent
(Hafner et al., 2021), which was tested to surpass (Hessel et al., 2018) and
other state-of-the-art model-free methods on the Atari 2600 benchmark.

Exploration methods Recent advances in exploration strategies for RL have
shown significant performance improvements in sparse-reward tasks. In the
following, we present the two methods that we tested in our experimental
setup:

  * •

ICM \- Intrinsic Curiosity Module (Pathak et al., 2017) consists of two
networks that enable computing a curiosity bonus for exploration. The first is
an inverse-dynamics model, used to learn compact features for reducing the
dimensionality of the observation inputs. The second network is a forward-
feature dynamics, which predicts the features of future observations by using
the features of the current observation and the current action. The curiosity
bonus is implemented as the error prediction of the forward-feature model,
computed as the Mean Squared Error (MSE) between predictions and true
features, and encourages the agent to explore different environment
transitions.

  * •

RND \- In Random Network Distillation (Burda et al., 2019b), they base upon
previous work showing that random features can perform well to reduce the
dimensionality of observations (Burda et al., 2019a). As a curiosity bonus,
they use the MSE prediction error between features computed by a randomly-
initialized network and a feature encoder they train, encouraging diversity in
the environment observations.

In our experiments, we add the intrinsic bonuses computed by ICM and RND to
the rewards, and train a PPO agent to maximize the compound returns.

##  Appendix B Experiment details

Since the environment is game-like, the decision was made to implement all
algorithms with their original parameters used for the Atari benchmark.

Algorithms that are benchmarked with Atari 2600 usually use preprocessed
images as input. More specifically, observations from the Atari games are
rescaled to 84x84 pixels and transformed to grayscale. For this reason, images
from our environment were preprocessed in the same manner.

The environment was initialized in the following way. The agent was given an
FOV of $1.1\pi$ radians (198 degrees) and a maximum amount of moves of 250.
The latter was chosen in a way to control the sparsity. Too many moves could
make the environment too easy to solve, so any random agent could obtain
reward 50% of the time. Too few moves could make the environment too difficult
to solve. Since the minimum amount of moves necessary to solve the environment
is around 50, the number was set to 250.

Important hyperparameters that may change between implementations in the
literature are shown in Table 1. Parameters that were adjusted for our
environment are emphasized. Exploration methods were tested in combination
with PPO. The exploration algorithms were switched off after the agent reached
20% average performance. Aside from the default PPO initialization method, we
ran a number of PPO experiments by initializing the policy network using large
orthogonal weights.

Table 1: Important hyperparameters in algorithm implementations. Some
hyperparameters may change between implementation found in literature. This
table indicates the exact parameters used. _Emphasized text_ indicates
variables that were adjusted by us. DQN
---
optimizer | RMSProp
learning rate | 2.5e-4
_steps per batch_ | 2500
min. replay history | 20000
batch size | 32
Rainbow
optimizer | Adam
learning rate | 6.25e-5
_steps per batch_ | 2500
min. replay history | 20000
batch size | 32
PPO
optimizer | Adam
learning rate | 2.5e-4
gradient clip | 0.1
value loss coefficient | 0.5
no. parallel agents | 8
_steps per batch_ | 250 (x8)
_no. mini batches_ | 8
entropy coefficient | 0.001
Exploration methods – PPO
exploration off threshold | 0.2