# Intrinsic Rewards for Exploration without Harm
from Observational Noise: A Simulation Study
Based on the Free Energy Principle

Theodore Jerome Tinker, Kenji Doya, Jun Tani∗

Abstract

In Reinforcement Learning (RL), artificial agents are trained to maximize
numerical rewards by performing tasks. Exploration is essential in RL because
agents must discover information before exploiting it. Two rewards encouraging
efficient exploration are the entropy of action policy and curiosity for
information gain. Entropy is well-established in literature, promoting
randomized action selection. Curiosity is defined in a broad variety of ways
in literature, promoting discovery of novel experiences. One example,
prediction error curiosity, rewards agents for discovering observations they
cannot accurately predict. However, such agents may be distracted by
unpredictable observational noises known as curiosity traps. Based on the Free
Energy Principle (FEP), this paper proposes hidden state curiosity, which
rewards agents by the KL divergence between the predictive prior and posterior
probabilities of latent variables. We trained six types of agents to navigate
mazes: baseline agents without rewards for entropy or curiosity, and agents
rewarded for entropy and/or either prediction error curiosity or hidden state
curiosity. We find entropy and curiosity result in efficient exploration,
especially both employed together. Notably, agents with hidden state curiosity
demonstrate resilience against curiosity traps, which hinder agents with
prediction error curiosity. This suggests implementing the FEP may enhance the
robustness and generalization of RL models, potentially aligning the learning
processes of artificial and biological agents.

##  1 Introduction

Reinforcement Learning (RL) is a machine learning algorithm for training
artificial agents to perform tasks by awarding or punishing their actions with
numerical rewards (Barto et al.,, 1983; Watkins and Dayan,, 1992; Mnih et
al.,, 2015). This can be interpreted as akin to biological agents learning
through evolution. Extrinsic rewards are awarded at human discretion based on
the tasks at hand. Intrinsic rewards are generated by agents themselves (with
human-provided hyperparameters) based on other goals such as exploration.
Exploration is an important but difficult aspect of RL because an agent can
only exploit knowledge after learning that knowledge. An exploration phase may
be implemented just by selecting random actions for the agent, but this can be
inefficient, especially with high-dimensional continuous state-action spaces
and sparse extrinsic rewards. Hence, we study two intrinsic rewards for
efficient exploration: entropy in the action-space for control as inference
(Millidge et al.,, 2020) and curiosity about the environment for active
inference (Tschantz et al.,, 2020, 2023). Meanwhile, Friston’s Free Energy
Principle (FEP) describes biological neuroscience with Bayesian statistics
applicable to machine learning and AI (Kaplan and Friston,, 2018; Parr and
Friston,, 2019). Our goal is to share a novel definition of curiosity derived
from the FEP and demonstrate its robust usefulness in exploration.

Curiosity in RL has been presented in many ways, typically based on an agent’s
ability to predict future observations. This leverages an adversarial
relationship between the agent’s predictive accuracy using a forward or
generative model, also known as a transitioner or world model, and its pursuit
of observations that challenge this model’s accuracy. For example,
Schmidhuber, (2010) critiqued that curiosity gauged by “mean squared
prediction error or similar measures,” which Oudeyer and Kaplan, (2007) called
Predictive Novelty Motivation and we call prediction error curiosity, “may
fail whenever high prediction errors do not imply expected prediction
progress, e.g., in noisy environments.” Alternative forms of curiosity from
these sources included estimating likelihoods of events, measuring prediction
errors probabilistically, or assessing improvement in predictions, but these
methods can have great computational costs. Moreover, Oudeyer and Kaplan note
“in certain application contexts… intrinsic openness is a weakness” and
counterproductive.

Pathak et al., (2017) elaborated on Schmidhuber’s critique of prediction error
curiosity in noisy environments, which may be an example of what Oudeyer and
Kaplan called a weakness of intrinsic openness. Pathak et al. asked readers to
consider an agent which could observe tree leaves randomly dancing in the
wind. The agent’s forward model would never be able to perfectly predict such
observations, so the agent might become fixated on these leaves like a moth
attracted to a lamp. Thus, such observational noises or expected uncertainties
are called curiosity traps. To remedy this, Pathak et al. trained an inverse
dynamics model to predict the agent’s action between two consecutive
observations, thereby encoding observations into a latent space without noisy
details, relevant only to the agent’s actions. Then, the agent’s forward model
could be trained to predict these refined latent states instead of chaotic
observations. Prediction error curiosity based on that forward model could
ignore the environment’s irrelevant noise and thus be minimally impacted by
curiosity traps, instead focusing on unexpected uncertainties. However, this
could not function well with rarely experienced interactions. Pathak et al.
suggested storing events in a memory buffer for experience replay, which we
implement.

Schwartenbeck et al., (2019) derived two intrinsic rewards for exploration
directly from the FEP. An agent with these intrinsic rewards uses Bayesian
inference to minimize free energy, meaning not only maximizing extrinsic
rewards but also developing a thorough understanding of the environment. The
first intrinsic reward, parameter exploration, encourages active learning: the
agent seeks to resolve uncertainty about how its actions are rewarded. The
second intrinsic reward, hidden state exploration, encourages active
inference: the agent seeks to take actions which reveal uncertain
observations. Together these intrinsic rewards establish curiosity for both
the environment and the task at hand. However, Schwartenbeck et al. assumed
agents in T-mazes knew the mazes had two arms, the left providing a constant
extrinsic reward, the right providing an uncertain extrinsic reward. Agents in
RL typically must infer such knowledge from observations, so these intrinsic
rewards are not easily applied to RL directly.

Kawahara et al., (2022) derived both entropy and an RL-applicable definition
of curiosity from the FEP. They constructed a forward model as a Bayesian
Neural Network (BNN) such that the weight-parameters are not fixed, but are
drawn from a multivariate Gaussian distribution using the reparameterization
trick for stochastic optimization (Blundell et al.,, 2015; Kingma et al.,,
2015). Kawahara et al. gauged curiosity values based on Kullback–Leibler
divergence comparing the weights’ distribution before and after learning to
predict each new observation with minimal free energy. This probabilistic
approach must consider uncertainty in observational noise, so like the
curiosity of Pathak et al., this curiosity should be able to effectively
explore without negative influence from curiosity traps. However, a BNN’s
computational cost is high, and Kawahara et al. only worked in terms of a
Markov Decision Process (MDP) where states are completely observed.
Additionally, this method produces just a single curiosity value per training
update, which, if applied to an entire batch, would only produce one curiosity
value for the batch as a whole, overlooking the contributions of each
observation individually. Identifying which parts of the batch are important
for exploration and which are not is a computationally demanding task because
each observation must be evaluated one at a time.

In this paper, we overcome the problem of prediction error curiosity using
hidden state curiosity defined in section 3. Like the curiosity of Kawahara et
al., hidden state curiosity is derived from the FEP and gauged by the
Kullback-Leibler divergence between predictive prior and posterior over future
states (under a particular policy). Like the curiosity of Pathak et al., those
states are efficiently encoded as latent variables. We train six types of
agents: a baseline with no intrinsic rewards, entropy-driven, prediction error
curious, hidden state curious, and two hybrids combining entropy with each
form of curiosity. The abilities of these agents to find goals in a biased
T-maze or an expanding T-maze (first a T-maze, then a double T-maze, and then
a triple T-maze) will test the following two hypotheses:

  1. 1.

Entropy and curiosity improve agent exploration, especially when both are
implemented together as implied by the FEP.

  2. 2.

Prediction error curiosity can be negatively influenced by observational noise
also known as curiosity traps, while hidden state curiosity can be more
resilient to such curiosity traps.

The results in section 4 are presented to evaluate the validity of these
hypotheses, contributing to our understanding of artificial intelligence
exploration in complex environments.

##  2 Prior Studies

###  2.1 Reinforcement Learning

In Reinforcement Learning (RL), an agent experiences an episode as a sequence
of transitions of the form $\\{o_{t},$ $a_{t},$ $r_{t},$ $o_{t+1},$
$done_{t}\\}$. Variables $o_{t}$ and $o_{t+1}$ are the agent’s observations at
times $t$ and $t+1$ equal to (in a Markov Decision Process, MDP) or derived
from (in a Partially Observable Markov Decision Process, POMDP) the complete
environmental states $s_{t}$ and $s_{t+1}$. Variable $a_{t}$ is the agent’s
action performed at time $t$. Variable $r_{t}$ is the extrinsic reward the
agent obtained by performing that action. And variable $done_{t}$ is 1 if time
$t$ was the final step in the episode or 0 otherwise.

In the Actor-Critic method (Barto et al.,, 1983), an agent has at least two
neural networks using parameters $\phi$ and $\theta$ to instantiate implicit
(i.e., amortise) mappings: an actor network $\pi_{\phi}(o_{t})\rightarrow
a_{t}$ also known as a policy, which chooses actions based on observations,
and a critic network $Q_{\theta}(o_{t},a_{t})\rightarrow\widehat{Q}_{t}$,
which predicts future rewards based on observations and actions to estimate
the state-action value function. The critic’s target value $Q_{t}$ is $r_{t}$
plus future rewards through bootstrapping (see equation 2.1, similar to
Bellman’s equation) predicted by a target critic $Q_{\bar{\theta}}$. The
target critic begins with parameters starting equal to the critic’s, then
slowly learns alongside the critic with Polyax averaging such that
$\bar{\theta}\leftarrow\tau\theta+(1-\tau)\bar{\theta}$ with hyperparameter
$\tau\in[0,1]$ being the soft update coefficient.

An ensemble of multiple critics (and an equal number of target critics) can be
implemented, in which case the actor’s loss function (equation 1) utilizes the
lowest predicted value among all critics. Furthermore, the actor’s training
can be staggered with a delay of $d$ epochs relative to the critics’ training,
ensuring less frequent actor training for stability. Regardless of these
implementations, the agent’s actor and critics shall be trained off-policy
with experience replay by randomly sampling a batch of transitions from a
memory buffer.

When using the Soft-Actor-Critic method (SAC) to implement entropy (Haarnoja
et al.,, 2018), the agent’s actions are randomly sampled with the
reparameterization trick for stochastic optimization (Kingma et al.,, 2015).
The actor generates two vectors the size of an action: mean $\mu_{t}$, made
without activation, and standard deviation $\sigma_{t}$, made with softplus
activation. The action is finally selected by combining these vectors with
noise $\epsilon_{t}$ sampled from the Unit Gaussian distribution $N(0,1)$,
resulting in $a_{t}=\tanh(\mu_{t}+\sigma_{t}\circ\epsilon_{t})$. Entropy in
action-choice is intrinsically rewarded in the actor’s loss function which
trains the actor to minimize $J_{\pi}$:

$J_{\pi}(\phi)=\mathbb{E}_{o_{t}\sim
D,a_{t}\sim\pi_{\phi}}\left[-Q_{\theta}(o_{t},a_{t})\right]-\alpha\mathcal{H}(\pi_{\phi}(a_{t}|o_{t}))$
(1)

with $D$ being the distribution of observations in the environment, non-
negative hyperparameter $\alpha$ being the relative importance of entropy, and
$\mathcal{H}(\pi_{\phi}(a_{t}|o_{t}))$ being the self-information (i.e.,
negative log probability) of an action $a_{t}$ given observation $o_{t}$,
whose expectation can be read as an action entropy. The hyperparameter
$\alpha$ can be chosen manually, or it can be a dynamic parameter which is
trained based on another hyperparameter, target entropy $\bar{\mathcal{H}}$.
In that case, $\alpha$ is chosen to minimize $J_{\alpha}$ in equation 2. If
the difference between entropy and target entropy is negative, actions are too
probable, so $\log(\alpha)$ will be increased to encourage more entropy; if
the difference is positive, actions are too improbable, so $\log(\alpha)$ will
be decreased to encourage less entropy.

$J_{\alpha}(\alpha)=\log(\alpha)\cdot(\mathcal{H}(\pi(a_{t}|o_{t}))-\bar{\mathcal{H}})$
(2)

Meanwhile, the SAC’s critic is trained to minimize $J_{Q}$ in the following
target value and loss function. We here also include an intrinsic reward for
curiosity, $P_{t}$, which can take several forms. In subsequent sections we
will consider the functional form of these alternatives and, in the final
section, use numerical experiments to quantify their contribution to optimal
behavior, as assessed via reward learning.

$\displaystyle Q(t)$ $\displaystyle=r_{t}+\eta P_{t}+$
$\displaystyle\gamma(1-done_{t})\mathbb{E}_{o_{t+1}\sim
D,a_{t+1}\sim\pi_{\phi}}[Q_{\bar{\theta}}(o_{t+1},a_{t+1})]-\alpha\mathcal{H}(\pi_{\phi}(a_{t+1}|o_{t+1}))$
(3)

$J_{Q}(\theta)=\mathbb{E}_{o_{t}\sim
D,a_{t}\sim\pi_{\phi}}[\left(Q_{\theta}(o_{t},a_{t})-Q(t)\right)^{2}]$ (4)

with non-negative hyperparameter $\eta$ being the relative importance of
curiosity value $P_{t}$ and hyperparameter $\gamma\in[0,1]$ being the
bootstrapping discount rate.

###  2.2 Recurrent RL

Yang and Nguyen, (2021) brought recurrent neural networks (RNN) to RL by
providing the actor and critic recurrent layers utilizing hidden states
$h^{\phi}_{t}$ and $h^{\theta}_{t}$ respectively (see figure 1(a)). Such an
actor or critic requires two additional inputs: the previous action $a_{t-1}$
and the previous hidden state $h^{\phi}_{t-1}$ or $h^{\theta}_{t-1}$ (in the
critic’s case, $h^{\theta}_{t-1}$ contains $a_{t-1}$ implicitly), with
$a_{-1}$, $h^{\phi}_{-1}$, and $h^{\theta}_{-1}$ being initialized as zero-
filled tensors. The actor’s hidden states (and, symmetrically, the critic’s
hidden states) would be calculated as

$h^{\phi}_{1},...,h^{\phi}_{t}=\text{RNN}_{\phi}(o_{1}||a_{0},...,o_{t}||a_{t-1})$

with $||$ denoting concatenation. To train recurrent models with experience
replay, a recurrent memory buffer must contain whole episodes beginning to end
for temporal context. Episodes may vary in length, so batches sampled from
recurrent memory buffers have all episodes standardized to the maximum
episode’s length using zero-filled transitions. Therefore, transitions of the
batch include another variable, $mask_{t}$, which is 1 if the transition was
actually within the episode or 0 if the transition was added for standard
length.

We utilize this method and another method described in section 3 depicted in
figure 1(b). Using either of these methods enables the agent to maintain a
continuity of experience, thus giving the agent a temporal edge in learning
and decision-making. This may be essential for navigating environments where
states are not directly observable.

Actor (RNN)$h^{\phi}_{t-1}$$a_{t-1}$$o_{t}$$h^{\phi}_{t}$$a_{t}$Equation 1ENV

Critic (RNN)$h^{\theta}_{t-1}$$a_{t}$$o_{t}$$h^{\theta}_{t}$QEquation 4

(a)

Actor (Forward’s Hidden State)$h^{q}_{t}$$a_{t}$Equation 1

Critic (Forward’s Hidden State)$h^{q}_{t}$$a_{t}$QEquation 4

(b)

Figure 1: (a) Implementing recurrent layers in an actor model and a critic
model. Notice the previous action $a_{t-1}$ is implicitly included in
$h^{\theta}_{t-1}$. (b) Implementing the forward model’s hidden state $h^{q}$
in an actor model and a critic model. Black arrows indicate forward
computations. Red arrows indicate loss functions for backpropagation.

###  2.3 Curiosity Derived from the FEP

Among various definitions of curiosity such as those from Oudeyer and Kaplan,
(2007), Schmidhuber, (2010), Pathak et al., (2017), and Schwartenbeck et al.,
(2019), the most relevant to this paper is that of Kawahara et al., (2022)
because it is directly derived from the FEP for RL. Here we first describe the
FEP as utilized by Kawahara et al., then describe their definition of
curiosity in that framework.

The cornerstone of the FEP is Bayes’ theorem, which describes the relationship
between the prior probability distribution of parameter $X$, $p(X)$, and the
posterior probability distribution given data $D$, $p(X|D)$, expressed as

$p(X|D)=\frac{p(D|X)p(X)}{p(D)}=\frac{p(D|X)p(X)}{\int p(D,X)dX}.$

However, $\int p(D,X)dX$ may be far too computationally expensive to
calculate. Instead we can approximate $p(X|D)$ with the predictive
distribution $q(X)$. The difference between $q(X)$ and $p(X|D)$ can be
measured using Kullback–Leibler divergence:

$\displaystyle D_{KL}[q(X)||p(X|D)]$ $\displaystyle=\int
q(X)\log\frac{q(X)}{p(X|D)}dX$ $\displaystyle=\int
q(X)\log\frac{q(X)p(D)}{p(D,X)}dX$ $\displaystyle=\int
q(X)\log\frac{q(X)p(D)}{p(X)p(D|X)}dX.$ (5)

Because that integral is in terms of $X$ and $p(D)$ is independent of $X$, we
may remove $p(D)$ from the integral with the laws of logarithms. This reveals
the concept of variational free energy $F$, defined as

$\displaystyle D_{KL}[q(X)||p(X|D)]$
$\displaystyle=\underbrace{D_{KL}[q(X)||p(X)]}_{\text{complexity}}-\underbrace{\mathbb{E}_{X}[\text{log}p(D|X)]}_{\text{accuracy}}+\text{
log }p(D)$ $\displaystyle=F+\log p(D).$ (6)

With this, we may estimate $q(X)$ as $\underset{q(X)}{\text{arg min }}F.$

Given time $\tau\geq t+1$, expected free energy $G_{\tau}$ is equal to free
energy $F$ multiplied by likelihood $P(D_{\tau}|X_{\tau})$, such that

$\displaystyle G_{\tau}$ $\displaystyle=\mathbb{E}_{p(D_{\tau}|X_{\tau})}[F]$
$\displaystyle=\mathbb{E}_{p(D_{\tau}|X_{\tau})}[\int
q(X_{\tau})\log\frac{q(X_{\tau})}{p(D_{\tau},X_{\tau})}dX]$
$\displaystyle=\mathbb{E}_{p(D_{\tau}|X_{\tau})}[\mathbb{E}_{q(X_{\tau})}[\log\frac{q(X_{\tau})}{p(X_{\tau}|D_{\tau})}-\log
p(D_{\tau})]].$ (7)

Because $q$ approximates $p$ and
$q(D_{\tau}|X_{\tau})q(D_{\tau})=q(D_{\tau},X_{\tau})$,

$\displaystyle G_{\tau}$
$\displaystyle\approx\mathbb{E}_{q(D_{\tau},X_{\tau})}[\log\frac{q(X_{\tau})}{q(X_{\tau}|D_{\tau})}-\log
p(D_{\tau})]$
$\displaystyle=-\mathbb{E}_{q(D_{\tau},X_{\tau})}[\log\frac{q(X_{\tau}|D_{\tau})}{q(X_{\tau})}]-\mathbb{E}_{q(D_{\tau})}[\log
p(D_{\tau})]$
$\displaystyle=-\underbrace{\mathbb{E}_{q(D_{\tau})}[\overbrace{D_{KL}[q(X_{\tau}|D_{\tau})||q(X_{\tau})]}^{\text{Bayesian
Surprise}}]}_{\text{Epistemic Value or Mutual
Information}}-\underbrace{\mathbb{E}_{q(D_{\tau})}[\log
p(D_{\tau})]}_{\text{Extrinsic Value}}.$ (8)

Kawahara et al., (2022) described how this equation can be used to interpret
biological agents’ behaviors. The Extrinsic Value term represents the
biological agents’ prior preferences or desires, much like extrinsic reward
$r_{t}$. The Epistemic Value term represents how living creatures choose
actions to find data $D$ providing information about parameter $X$, reducing
uncertainty. Hence, biological agents seek to satisfy their desires while also
acquiring unknown information.

Kawahara et al. applied the FEP to RL by training agents to choose actions
which minimize expected free energy, emulating biological thought. Let
$D_{\tau}=\\{s_{\tau},a_{\tau}\\}$ (with the understanding that Kawahara et
al. considered states to be completely observed such that $o_{\tau}=s_{\tau}$)
so expected free energy can be rewritten as

$\displaystyle G_{\tau}$
$\displaystyle=-\mathbb{E}_{q(s_{\tau},a_{\tau},X_{\tau})}[\log\frac{p(X_{\tau}|s_{\tau},a_{\tau})}{q(X_{\tau})}]-\mathbb{E}_{q(s_{\tau},a_{\tau})}[\log
p(s_{\tau},a_{\tau})]$
$\displaystyle=-\mathbb{E}_{q(s_{\tau},a_{\tau},X_{\tau})}[\log\frac{p(X_{\tau},a_{\tau}|s_{\tau})}{q(X_{\tau})p(a_{\tau}|s_{\tau})}]-\mathbb{E}_{q(s_{\tau},a_{\tau})}[\log
p(s_{\tau},a_{\tau})]$
$\displaystyle\approx-\mathbb{E}_{q(s_{\tau},a_{\tau},X_{\tau})}[\log\frac{q(X_{\tau}|s_{\tau})q(a_{\tau}|s_{\tau},X_{\tau})}{q(X_{\tau})p(a_{\tau}|s_{\tau})}]-\mathbb{E}_{q(s_{\tau},a_{\tau})}[\log
p(s_{\tau},a_{\tau})]$
$\displaystyle=-\mathbb{E}_{q(a_{\tau}|s_{\tau},X_{\tau})q(s_{\tau})}[D_{KL}[q(X_{\tau}|s_{\tau})||q(X_{\tau})]]$
$\displaystyle-\mathbb{E}_{q(s_{\tau},X_{\tau})}[D_{KL}[q(a_{\tau}|s_{\tau},X_{\tau})||p(a_{\tau}|s_{\tau})]]$
$\displaystyle-\mathbb{E}_{q(s_{\tau},a_{\tau})}[\log p(s_{\tau},a_{\tau})].$
(9)

Consider now a forward model $f_{w}$ which is a Bayesian Neural Network (BNN)
(Blundell et al.,, 2015). Because this forward model predicts the next state
given the current state and action, its weight parameters $w_{\tau}$ can be
interpreted as the agent’s latent understanding of the environment; thus, let
$X_{\tau}=w_{\tau}$ such that the first term of equation 2.3 reflects how the
agent’s understanding of the environment changes based on experiencing a
state. Let $q_{\psi}=\mathcal{N}(w_{\tau}|\mu,\sigma)$ with
$\psi=\\{\mu,\sigma\\}$, so a Soft Actor Critic actor $\pi_{\phi}$ can be
trained to approximate $\pi_{\phi}(a_{\tau}|s_{\tau})\approx
q(a_{\tau}|s_{\tau},w_{\tau})$. This allows rewriting expected free energy as

$\displaystyle G(s_{\tau},a_{\tau})$
$\displaystyle=-\mathbb{E}_{q(a_{\tau}|s_{\tau},w_{\tau})q(s_{\tau})}[D_{KL}[q(w_{\tau}|s_{\tau})||q(w_{\tau})]]$
$\displaystyle-\mathbb{E}_{q(s_{\tau},a_{\tau})}[D_{KL}[\pi_{\phi}(a_{\tau}|s_{\tau})||p(a_{\tau}|s_{\tau})]]$
$\displaystyle-\mathbb{E}_{q(s_{\tau},a_{\tau})}[\log p(s_{\tau},a_{\tau})].$
(10)

Finally, let $\tau=t$, let prior preference extrinsic value $\log
p(s_{t},a_{t})$ be extrinsic reward $r(s_{t},a_{t})=r_{t}$, and recall the
forward model’s objective is predicting $s_{t+1}$, so expected free energy can
be rewritten as

$\displaystyle G(s_{t},a_{t})$
$\displaystyle=-D_{KL}[q_{\psi}(w_{t}|s_{t+1})||q_{\psi}(w_{t})]-\log
p(s_{t},a_{t})$ $\displaystyle-
D_{KL}[\pi_{\phi}(a_{t}|s_{t})||p(a_{t}|s_{t})]$
$\displaystyle=-D_{KL}[q_{\psi}(w_{t}|s_{t+1})||q_{\psi}(w_{t})]-\log
p(s_{t},a_{t})$
$\displaystyle-\int\pi_{\phi}(a_{t}|s_{t})\log\pi_{\phi}(a_{t}|s_{t})da_{t}+\int\pi_{\phi}(a_{t}|s_{t})\log
p(a_{t}|s_{t})da_{t}$
$\displaystyle=-\underbrace{D_{KL}[q_{\psi}(w_{t}|s_{t+1})||q_{\psi}(w_{t})]}_{\text{Curiosity}}-\underbrace{r(s_{t},a_{t})}_{\text{Extrinsic
Reward}}$
$\displaystyle-\underbrace{\mathcal{H}(\pi_{\phi}(a_{t}|s_{t}))}_{\text{Entropy}}-\underbrace{\mathbb{E}_{\pi_{\phi}(a_{t}|s_{t})}[\log
p(a_{t}|s_{t})]}_{\text{Imitation}}.$ (11)

In this formation, the RL implementation of the FEP presents extrinsic
rewards, entropy, and curiosity. (Imitation, an intrinsic reward which we do
not explore in this paper, could further be used to train an agent to emulate
the behavior of another agent.) Equations 1 and 2.1 can incorporate these
entropy and curiosity values or approximations thereof, multiplied by non-
negative hyperparameters $\alpha$ and $\eta$. Calculating the curiosity value
$P_{t}=D_{KL}[q_{\psi}(w_{t}|s_{t+1})||q_{\psi}(w_{t})]$ involves two
distributions: $q_{\psi}(w_{t})$, the distribution of $f_{w}$’s weights, and
$q_{\psi}(w_{t}|s_{t+1})$, the same distribution after considering the next
state. The first distribution, $q_{\psi}(w_{t})$, is readily available; it may
be called a predictive prior. We can then obtain the predictive posterior
distribution $q_{\psi}(w_{t}|s_{t+1})$ as if training the forward model to
predict $s_{t+1}$ with minimal free energy $F$, calculated as

$F=\underbrace{D_{KL}[q_{\psi}(w_{t})||p_{\psi}(w_{t})]}_{\text{Complexity}}-\underbrace{\mathbb{E}_{q_{\psi}(w_{t})}[\log
p_{\psi}(s_{t+1}|w_{t})]}_{\text{Accuracy}}.$ (12)

In practice, prior distribution $p_{\psi}(w_{t})$ can be the unit Gaussian
$\mathcal{N}(0,1)$, and the complexity term is multiplied by a non-negative
hyperparameter $\beta$ describing its relative importance to accuracy.

Thusly, Kawahara et al. used the FEP’s framework to define curiosity such that
an observation has a low curiosity value if the forward model does not need to
change much to accommodate it, or a high curiosity value if the forward model
must change drastically. This encourages an adversarial relationship between
the agent’s forward model and actor: the forward model trains to improve its
weights representing a probabilistic interpretation of the environment, but
the critic rewards the actor for finding information which substantially
alters the forward model’s weights. This active inference complements
entropy’s control as inference. Importantly, the agent’s probabilistic
interpretation of the environment should account for observational noise, so
observing anticipated noise should not alter it much; hence, the free energy
based curiosity defined by Kawahara et al. should be able to effectively
explore without negative influence from curiosity traps. However, this
definition of curiosity is constrained regarding batch processing: individual
transitions within a batch may differ in exploratory importance, but comparing
the forward model’s weights before and after training with the entire batch as
a whole returns only one curiosity value. Investigating the significance of
each transition individually requires great computational cost. That
limitation, and the restriction to fully observable MDP, suggest there are
opportunities for further development.

##  3 Proposed Model

Motivated by Kawahara et al. and Pathak et al., we define hidden state
curiosity using a forward model with the architecture of a Variational RNN
(VRNN) (Chung et al.,, 2016). Our forward model $f_{\psi}$, pictured in figure
2 and described by algorithm 1, is recurrent using hidden state $h^{q}_{t}$,
enabling the accounting of temporal dependencies and uncertainties in a
Partially Observable Markov Decision Process (POMDP). This provides a second
method for providing the actor and critic temporal knowledge: replacing
observations in the models’ inputs with $h^{q}_{t}$ (see figure 1(b)). We
apply this method to the actor, with a training-delay of $d=2$, while two
critics use their own recurrent layers, as shown in figure 1(a) (see section
2.1 regarding delayed actor training and multiple critics). Unlike the
Bayesian Neural Network (BNN) used by Kawahara et al., our forward model does
not have probabilistic weights; instead, we use the reparameterization trick
in the style of a SAC actor or Variational Bayes Autoencoder (VAE) to sample
prior and posterior inner states $z^{p}_{t}$ and $z^{q}_{t}$ from
corresponding probability distributions
$p(z_{t})=\mathcal{N}(\mu^{p}_{t},\sigma^{p}_{t})$ and
$q(z_{t})=\mathcal{N}(\mu^{q}_{t},\sigma^{q}_{t})$. These distributions are
derived from the previous hidden state $h^{q}_{t-1}$, the previous action
$a_{t-1}$, and in the case of the posterior inner state, the current
observation $o_{t}$. Posterior inner state $z^{q}_{t}$ and previous hidden
state $h^{q}_{t-1}$ are used to generate hidden state $h^{q}_{t}$.

InitiateStep
0Forward$h^{q}_{-1}=0.0$$a_{-1}=0.0$$\mathcal{N}(\mu^{p}_{0},\sigma^{p}_{0})$
$=p(z_{0})$$z^{p}_{0}$$\mathcal{N}(\mu^{q}_{0},\sigma^{q}_{0})$
$=q(z_{0})$$z^{q}_{0}$$o_{0}$$h^{q}_{0}$$a_{0}$$\widehat{o}_{1}$$o_{1}$ComplexityAccuracy
Figure 2: Forward model’s architecture based on VRNN. Black arrows indicate
forward computations. Red arrows indicate errors for backpropagation.
Algorithm 1 Pseudocode for Proposed Model (Part One)

Initialize forward model $f_{\psi}$, actor $\pi_{\phi}$, critic $Q_{\theta}$,
replay buffer $R$ $\triangleright$ Two critics used

Initialize target critic weights $\bar{\theta}\leftarrow\theta$
$\triangleright$ One target critic for each critic

for epoch = 0, M do

$\triangleright$ In each epoch, the agent plays one episode and trains with a
batch of episodes

Initialize $h^{q}_{-1}$ and $a_{-1}=0.0$, receive $o_{0}$ $\triangleright$
Begin new episode

for t = 0, T do $\triangleright$ Steps in episode

$\mu^{q}_{t},\sigma^{q}_{t}\leftarrow\textrm{MLP}(h^{q}_{t-1}||a_{t-1}||o_{t})$

$\triangleright$ Posterior inner state distribution; $||$ denotes
concatenation

$z^{q}_{t}\sim q(z_{t})=\mathcal{N}(\mu^{q}_{t},\sigma^{q}_{t}))$
$\triangleright$ Sample posterior inner state

$h^{q}_{t}\leftarrow\textrm{RNN}(h^{q}_{t-1},z^{q}_{t})$ $\triangleright$
Advance $h^{q}$

Execute action $a_{t}\leftarrow\pi_{\phi}(a_{t}|h^{q}_{t})$ to receive
$o_{t+1}$, $r_{t}$, and $done_{t}$

$\triangleright$ If $done_{t}$, stop episode

end for

Store episode’s transitions $(o_{0:T+1},a_{-1:T},r_{0:T},done_{0:T})$ in $R$
$\triangleright$ Save episode

Sample batch of episodes $(o_{0:T+1},a_{-1:T},r_{0:T},done_{0:T},mask_{0:T})$
from $R$

$\triangleright$ Sample batch

Initialize $h^{q}_{-1}$, $h^{\theta}_{-1}$, and $h^{\prime\theta}_{-1}=0.0$
$\triangleright$ Begin training

for t = 0, T+1 do $\triangleright$ Steps in advancing forward model with batch

$\mu^{p}_{t},\sigma^{p}_{t}\leftarrow\textrm{MLP}(h^{q}_{t-1}||a_{t-1})$
$\triangleright$ Prior inner state distribution

$\mu^{q}_{t},\sigma^{q}_{t}\leftarrow\textrm{MLP}(h^{q}_{t-1}||a_{t-1}||o_{t})$
$\triangleright$ Posterior inner state distribution

Note: Algorithm continues. See sections 2.1 and 2.2 for details on $o_{t}$,
$a_{t}$, $r_{t}$, $done_{t}$, $mask_{t}$, delayed actor, and multiple critics.

Algorithm 2 Pseudocode for Proposed Model (Part 2)

if utilizing hidden state curiosity then

$P_{t-1}\leftarrow D_{KL}[q(z_{t})||p(z_{t})]$

$\triangleright$ Compare prior and posterior for hidden state curiosity

else

$P_{t}\leftarrow\mathbb{E}_{q(z_{t})}[\log p(o_{t+1}|z_{t})]$

$\triangleright$ Evaluate prediction for prediction error curiosity

end if

$z^{q}_{t}\sim q(z_{t})=\mathcal{N}(\mu^{q}_{t},\sigma^{q}_{t}))$
$\triangleright$ Sample posterior inner state

$h^{q}_{t}\leftarrow\textrm{RNN}(h^{q}_{t-1},z^{q}_{t})$ $\triangleright$
Advance $h^{q}$

end for

for t = 0, T+1 do $\triangleright$ Steps in advancing critics with batch

$\widehat{Q}_{t},h^{\theta}_{t}\leftarrow
Q_{\theta}(o_{t},a_{t},h^{\theta}_{t-1})$ $\triangleright$ Predict $Q$-value
and advance $h^{\theta}$ with both critics

$h^{\prime\theta}_{t}\leftarrow
Q_{\bar{\theta}}(o_{t},a_{t},h^{\theta}_{t-1})$ $\triangleright$ Advance
$h^{\prime\theta}$ for both target critics (ignore $Q$ output)

$a^{\prime}_{t}\leftarrow\pi_{\phi}(a_{t}|h^{q}_{t})$ $\triangleright$ Make
new action with actor

if $t>1$ then $\triangleright$ After first step, make target $Q$-values with
both target critics

$Q_{t}\leftarrow Q_{\bar{\theta}}(o_{t},a^{\prime}_{t},h^{\theta}_{t-1})$

$\triangleright$ Get target critics’ $Q$-values (ignore $h^{\prime\theta}$
output)

$Q_{t}\leftarrow r_{t}+\eta
P_{t}+\gamma(1-done_{t})(Q_{t}-\alpha\mathcal{H}(\pi_{\phi}(a^{\prime}_{t}|o_{t})))$

$\triangleright$ Make target $Q$-values (eq. 2.1)

end if

end for

Note: Algorithm continues.

Algorithm 3 Pseudocode for Proposed Model (Part 3)

$F\leftarrow(\beta D_{KL}[q(z_{0:T})||p(z_{0:T})]-\mathbb{E}_{q(z_{0:T})}[\log
p(o_{1:T+1}|z_{0:T})])*mask_{0:T}$

$\triangleright$ Forward model loss (free energy, eq. 13)

$\psi\leftarrow\psi-\lambda_{\psi}\frac{\partial F}{\partial\psi}$
$\triangleright$ Train forward model

$J_{Q}(\theta)\leftarrow(\widehat{Q}_{0:T}-Q_{1:T+1})^{2}*mask_{0:T}$
$\triangleright$ Critic loss (eq. 4)

$\theta\leftarrow\theta-\lambda_{Q}\hat{\nabla}_{\theta}J_{Q}(\theta)$
$\triangleright$ Train critics

$\bar{\theta}\leftarrow\tau\theta+(1-\tau)\bar{\theta}$ $\triangleright$
Update target critics

if utilizing dynamic $\alpha$ with target entropy $\mathcal{\bar{H}}$ then

$J_{\alpha}(\alpha)\leftarrow\log(\alpha)\cdot(\mathcal{H}(\pi(a_{0:T}|o_{0:T}))-\bar{\mathcal{H}})*mask_{0:T}$
$\triangleright$ Alpha loss (eq. 2)

$\log(\alpha)\leftarrow\log(\alpha)-\lambda_{\alpha}\hat{\nabla}_{\alpha}J_{\alpha}(\alpha)$
$\triangleright$ Train alpha

end if

if $\textrm{epoch}\equiv 0\textrm{ mod }d$ then

$\triangleright$ In each $d^{\textrm{th}}$ epoch, train actor; use lowest
$Q$-value among critics

$J_{\pi}(\phi)\leftarrow(-Q_{\theta}(o_{0:T},a^{\prime}_{0:T})-\alpha\mathcal{H}(\pi_{\phi}(a^{\prime}_{0:T}|o_{0:T})))*mask_{0:T}$

$\triangleright$ Actor loss (eq. 1)

$\phi\leftarrow\phi-\lambda_{\pi}\hat{\nabla}_{\phi}J_{\pi}(\phi)$
$\triangleright$ Train actor

end if

end for

Note: Algorithm is completed.

This forward model trains to generate accurate predictions $\widehat{o}_{t+1}$
of upcoming observations $o_{t+1}$ with hidden states $h^{q}_{t}$ and actions
$a_{t}$ while minimizing the Kullback-Leibler divergence comparing $p(z_{t})$
and $q(z_{t})$. The relationship between $p(z_{t})$ and $q(z_{t})$ is parallel
to the relationship between Kawahara et al.’s $q(w_{t})$ and $q(w_{t}|s_{t})$,
used in equation 2.3 to replace $q(X_{t})$ and $q(X_{t}|s_{t})$ in equation
2.3: these pairs are an agent’s probabilistic understanding of its environment
before and after experiencing a state or observation. Thus, this forward model
is trained to minimize free energy $F$ with equation 12 rewritten as equation
13.

$\displaystyle
F=\underbrace{D_{KL}[q(z_{t})||p(z_{t})]}_{\text{Complexity}}-\underbrace{\mathbb{E}_{q(z_{t})}[\log
p(o_{t+1}|z_{t})]}_{\text{Accuracy}}$ (13)

In practice, the complexity term is multiplied by a non-negative
hyperparameter $\beta$ describing its relative importance to accuracy.

Notice that the curiosity term in Equation 2.3 and the complexity term in
Equation 12 are both KL divergences that quantify information gains (a.k.a.,
relative entropies). Inferring latent states or parameters—by minimizing
variational free energy—minimizes the divergence between the posterior and
prior, given an observation; thereby minimizing complexity and implicitly
maximizing compression and generalization. Conversely, during action
selection, based upon expected free energy, the divergence between the
predictive posterior and prior is maximized, to maximize expected information
gain. One can leverage these information theoretic interpretations by using
the complexity term as an estimate of the expected information gain, under the
policy being learned. This is licensed because the actor-critic model used in
reinforcement learning learns a state-action policy that can, effectively,
learn the expected information gain in the same way that it learns expected
extrinsic rewards. In other words, one can define hidden state curiosity at
the previous time step to be the complexity in Equation 14. (In practice, we
apply a clamp constraining hidden state curiosity between 0 and 1). Thus, the
actor and critics are trained to minimize expected free energy, rewriting
equation 2.3 as

$\displaystyle G(s_{t},a_{t})$
$\displaystyle=-\underbrace{D_{KL}[q(z_{t+1})||p(z_{t+1})]}_{\text{Hidden
State Curiosity}}-\underbrace{r(s_{t},a_{t})}_{\text{Extrinsic Reward}}$
$\displaystyle-\underbrace{\mathcal{H}(\pi_{\phi}(a_{t}|o_{t}))}_{\text{Entropy}}-\underbrace{\mathbb{E}_{\pi_{\phi}(a_{t}|o_{t})}[\log
p(a_{t}|o_{t})]}_{\text{Imitation}}.$ (14)

Equations 1 and 2.1 can incorporate approximations of these entropy and
curiosity values, multiplied by non-negative hyperparameters $\alpha$ and
$\eta$ (not $\beta$) respectively. This underwrites an adversarial
relationship: the forward model is trained to make accurate predictions with
minimal complexity but the critic rewards the actor for choosing actions that
result in high complexity; namely, a high information gain that characterizes
curious or information seeking policies. This is in contrast to prediction
error curiosity, which rewards the actor for choosing actions resulting in
poor accuracy. No agent can perfectly predict an observation which includes
random noise, leaving accuracy perpetually flawed, so agents with prediction
error curiosity will always be curious about such an observation. However, an
agent employing hidden state curiosity may generate a prior inner state which
foresees the observation’s random noise; thus, the random noise might not
provide any additional information for the posterior inner state, so the
agent’s prior and posterior conceptualization of the environment can perfectly
match despite that randomness. Therefore, rather than seeking merely noisy
observations, such agents seek observations which actually alter their latent
interpretations of the environment. This reflects how Pathak et al. utilized
latent spaces ignoring irrelevant noise, so we predict hidden state curiosity
can effectively explore without negative influence from curiosity traps. Also,
$p(z_{t})$ and $q(z_{t})$ can be newly generated for each individual
transition in a batch of episodes with relatively easy computational cost,
assigning curiosity values in a scalable manner.

##  4 Simulation Experiments

As established in section 1, we designed our experiments to investigate these
two hypotheses:

  1. 1.

Entropy and curiosity improve agent exploration, especially when both are
implemented together as implied by the FEP.

  2. 2.

Prediction error curiosity can be negatively influenced by observational noise
also known as curiosity traps, while hidden state curiosity can be more
resilient to such curiosity traps.

To this end, our experiments feature six types of agents training to find
goals in various mazes. These will be baseline agents devoid of intrinsic
rewards, agents motivated by either entropy or one form of curiosity
(prediction error or hidden state), and agents motivated by a combination of
entropy and one type of curiosity. See table 1 for details about these six
types. All agents will share the same architecture with $\gamma=.9$,
$\tau=.1$, $d=2$, $\bar{\mathcal{H}}=-1$ (see section 2.1), $\beta=.03$ (see
equation 13), and learning rate $\lambda=.01$ with Adam optimizers. No agents
have periods of forced investigation with random actions, highlighting the
importance of motivating exploration. See tables 2 through 8 in appendix B for
details about models’ architectures in PyTorch. Each table illustrates the
parameters of a model layer by layer.

Name (and Acronym) | $\alpha$ | $\eta$ | $P_{t}$
---|---|---|---
No Entropy, No Curiosity (N) | 0 | 0 | None
Entropy (E) | None | 0 | None
Prediction Error Curiosity (P) | 0 | 1 | $\mathbb{E}_{q(z_{t})}[\log p(o_{t+1}|z_{t})]$
Entropy and Prediction Error Curiosity (EP) | None | 1 | $\mathbb{E}_{q(z_{t})}[\log p(o_{t+1}|z_{t})]$
Hidden State Curiosity (H) | 0 | 1 | $D_{KL}[q(z_{t+1})||p(z_{t+1})]$
Entropy and Hidden State Curiosity (EH) | None | 1 | $D_{KL}[q(z_{t+1})||p(z_{t+1})]$
Table 1: Hyperparameters for six types of agents. Recall $\alpha$ and $\eta$
in equations 1 and 2.1. If $\alpha=$ None, then $\alpha$ is a dynamic
parameter chosen to minimize equation 2. In Prediction Error curiosity (P and
EP), the curiosity value $P_{t}$ is the accuracy term of equation 13. In
Hidden State Curiosity (H and EH), $P_{t}$ is the curiosity term of equation
14.

Regarding the first hypothesis, we predict the baseline agent will perform the
least efficient exploration, while agents rewarded for both entropy and
curiosity will outperform the rest, regardless of which kind of curiosity.
However, regarding the second hypothesis, we predict that if we train agents
in mazes with curiosity traps, agents with prediction error curiosity will be
attracted to those traps, showcasing its susceptibility, while agents with
hidden state curiosity are able to ignore them.

###  4.1 Experiment Design

In these experiments, we employ the PyBullet physics engine to simulate an RL
agent embodied as a duck. The agent’s observations have two parts: its current
speed and an 8 by 8 by 4 image of what is in front of it with the four
channels being red, green, blue, and distance. The agent’s actions also have
two parts: adjusting its yaw up to 90 degrees left or right and choosing a
speed between 0 meters per time step and a speed limit (with the blocks
constructing the mazes having side-length of one meter).

Figure 3: An agent’s observation includes its current speed in meters per time
step and an 8 by 8 by 4 image of what is in front of it. The image’s four
channels are red, green, blue (left), and distance (right). This is the
agent’s first observation in the biased T-maze; see figure 4(a).

Each simulated episode is terminated when the agent exits the maze; the
agent’s choice of exit will earn an extrinsic reward or punishment. If no exit
is chosen within 30 steps, the episode ends with a punishment of $r=-1$.
Colliding with a maze wall at any step will also punish the agent with $r=-1$.
Any positive extrinsic rewards are multiplied by $.99^{\text{steps taken}}$,
encouraging haste.

In each epoch, the agent will carry out one episode. Memory of that episode’s
transitions will be saved in that agent’s recurrent replay buffer; if that
replay buffer contains memory of more than 250 episodes, the oldest episode
will be deleted. Then a batch of 32 episodes will be sampled from the replay
buffer to train the agent’s forward model, actor, and critics. In this manner,
we will use different random seeds to train 360 agents of each of the six
types described above. This will be carried out both with and without
implementing curiosity traps by randomly changing colors of walls near
inferior exits with every step, investigating which types of agents are
disadvantaged by observational noise.

(a) Biased T-Maze.

(b) T-Maze.

(c) Double T-Maze.

(d) Triple T-Maze.

Figure 4: Agent starts where shown. Correct and incorrect exits are marked ✓
and X. With curiosity traps, blocks with ? change colors each step. Experiment
one uses (a) biased T-Maze. Experiment two uses (b) T-maze, (c) double T-maze,
(d) triple T-maze.

####  4.1.1 Biased T-Maze

In the biased T-maze simulation seen in figure 4(a), agents have a speed limit
of one meter per time step. The biased T-maze has two exits. The exit out of
the T’s left arm is easily accessible (nearby and unobstructed) and provides
an extrinsic reward of $r=1$. The exit out of the T’s right arm is difficult
to access (farther away and behind an obstacle) and provides an inconsistent
extrinsic reward which is equally likely to be $r=0$ or $r=10$ with an
expected value $\mathbb{E}(r)=5$. Intuitively for human readers, the option
with highest expected extrinsic value is the exit to the right despite its
distance, so we will call this the correct exit. An agent, however, must
explore to the right despite readily available reward on the left to discover
the higher value of the correct exit. We trained agents for 500 epochs using
the six sets of hyperparameters described in table 1 with or without curiosity
traps near the incorrect exit.

We predict agents will discover and exploit the higher value of the correct
exit more often when encouraged with entropy or curiosity, especially both at
once. We also expect curiosity traps to negatively impact performance of
agents with prediction error curiosity, while agents with hidden state
curiosity are able to ignore them.

####  4.1.2 Expanding T-Maze

In the expanding T-maze simulation, agents have a speed limit of two meters
per time step, first in the T-maze seen in figure 4(b), then the double T-maze
seen in figure 4(c), and then the triple T-maze seen in figure 4(d). In each
of the three mazes, only one exit is deemed correct, with its location
alternating between successive mazes to challenge the agents’ ability to
override previously learned habitual behaviors. If the agent takes the correct
exit, it will be rewarded with $r=10$, but if the agent takes any other exit,
it will be punished with $r=-.5$. We trained agents for 500 epochs in the
T-maze, then for 2000 epochs in the double T-maze, and then for 4000 epochs in
the triple T-maze. We trained agents using the six sets of hyperparameters
described in table 1 with or without curiosity traps near the incorrect exits
in the mazes’ bottom left portions.

We predict all agents, even those without intrinsic rewards for exploration,
to easily discover and exploit the correct exit on the right side of the
T-maze. Then, when relocated to the double T-maze, we predict all agents to
first move to the right, away from the correct exit now on the left side. We
predict agents without intrinsic rewards for exploration will have difficulty
extinguishing that learned behavior, while agents intrinsically rewarded with
entropy or curiosity, especially both, are able to begin exploring again to
find the new correct exit. Finally, relocation to the triple T-maze will
present this challenge again on a larger scale. We also predict curiosity
traps to negatively impact performance of agents with prediction error
curiosity, while agents with hidden state curiosity are able to ignore them.

###  4.2 Results

In figures 5 and 6, see the trajectories of agents trained in the biased
T-maze and expanding T-maze, depicting their behaviors. Find an example of an
agent’s forward model predicting observations in the biased T-maze in figure
7. In figures 10 and 11, in appendix A, see the proportions of agents choosing
each exit in each epoch. Find videos of how agent trajectories changed over
time at github.com/oist-cnru/curious_maze.

Figure 5: Trajectories of agents after training in the biased T-maze. The
correct exit of each maze is marked with a ✓, while each incorrect exit is
marked with an X. If curiosity traps are applied, blocks marked with a ? will
change to random colors with every step. Figure 6: Trajectories of agents
after training in the expanding T-maze. The correct exits are marked with a ✓,
while incorrect exits are marked with an X. If curiosity traps are applied,
blocks marked with a ? will change to random colors with every step. Figure 7:
Predictions of agent choosing correct exit in biased T-maze, trained with EH
(see table 1) without curiosity traps. Left column: actual observations.
Middle column: predictions based on hidden-state $h^{p}$, made with prior
inner state $z^{p}$. Right column: predictions based on hidden-state $h^{q}$,
made with posterior inner state $z^{q}$.

Statistic results of these experiments are displayed in figures 8 and 9, which
show how often agents trained with the hyperparameters described in table 1
reached the correct exit in their final ten episodes.

Figures 8(a) and 9(a) regard our first hypothesis: entropy and curiosity
improve agent exploration, especially when both are implemented together as
implied by the FEP.

Figures 8(b) and 9(b) regard our second hypothesis: prediction error curiosity
can be negatively influenced by observational noise also known as curiosity
traps, while hidden state curiosity can be more resilient to such curiosity
traps.

####  4.2.1 Biased T-Maze

In figure 8(a), the leftmost bar shows that agents trained using the
hyperparameters labeled “No Entropy, No Curiosity” (acronym N) were the least
successful agents in the biased T-maze. The three bars between the dotted
lines show that agents trained using “Entropy” (E), “Prediction Error
Curiosity” (P), or “Hidden State Curiosity” (H) all performed as well as or
better than N utilizing one intrinsic reward. The rightmost bars show agents
trained with two intrinsic rewards using “Entropy and Prediction Error
Curiosity” (EP) or “Entropy and Hidden State Curiosity” (EH) performed best of
all, demonstrating the importance of combining these intrinsic rewards.

In figure 8(b), to the left of the dotted line, note that agents trained using
P or EP performed significantly worse when trained with curiosity traps. In
contrast, right of the dotted line, agents trained using H or EH have no
negative impact from curiosity traps. This demonstrates that hidden state
curiosity can mitigate pitfalls which can entrap prediction error curiosity.

These results can be visually confirmed in figures 5 and 10. In figure 5, some
agents trained using EP with curiosity traps only travel in circles to fixate
on the randomly changing walls, revealing the distraction caused by
observational noise. Figure 10, in appendix A, shows how agents trained using
N commit to the first exit they encounter, as the rates of exit-choice
increase epoch to epoch but never decrease. In contrast, other agents select
the incorrect exit at an increasing rate until a peak at approximately the
$100^{\textrm{th}}$ epoch, at which point selection of the correct exit
increases instead, as if the agents became bored with the easy exit and
explored instead.

####  4.2.2 Expanding T-Maze

Figure 9(a) displays the performance of agents at the end of training in the
T-maze, then the double T-maze, and finally the triple T-maze. Consistent to
all three mazes, and just like in the biased T-maze, the leftmost bar shows
that agents trained using the hyperparameters labeled “No Entropy, No
Curiosity” (acronym N) were the least successful agents. The three bars
between the dotted lines show that agents trained using “Entropy” (E),
“Prediction Error Curiosity” (P), or “Hidden State Curiosity” (H) all
performed as well as or better than N with one intrinsic reward. The rightmost
bars show agents trained with two intrinsic rewards using “Entropy and
Prediction Error Curiosity” (EP) or “Entropy and Hidden State Curiosity” (EH)
performed best of all, demonstrating the importance of combining these
intrinsic rewards.

In figure 9(b), to the left of the dotted line, note that agents trained using
P performed worse when trained with curiosity traps in all three of these
mazes, and agents trained using EP were deeply influenced by curiosity traps
in the T-maze and triple T-maze. In contrast, right of the dotted line, agents
trained using H or EH have no negative impact from curiosity traps. This
demonstrates that hidden state curiosity can mitigate pitfalls which can
entrap prediction error curiosity.

These results can be visually confirmed in the figures 6 and 11. In figure 6,
the impact of curiosity traps is clear when some agents trained using P or EP
with curiosity traps are attracted to randomly colored walls in all three of
the T-mazes. In figure 11, in appendix A, we see that many agents trained
using N learned to reach the correct exit to the right on the T-maze, but many
of them continued to select exits on the right side when relocated into the
double T-maze even though the correct exit was now on the left. Likewise, many
agents trained using E learned to reach the correct exit in the double T-maze
with a left turn and then a right turn, but continued to select exits on the
left side when relocated into the triple T-maze even through the correct exit
was now on the right. In contrast, agents trained using H or EH swiftly
stopped choosing incorrect exits when relocated, whether with or without
curiosity traps.

(a)

(b)

Figure 8: Biased T-Maze Results. (a) Bars show rate agents chose the correct
exit in their last ten episodes after training using hyperparameters labeled
with acronyms from table 1. Error bars show 99% confidence interval. (b) In
each pair of bars, if the left bar is taller than the right bar with
confidence of 99%, the left bar is green and the right bar is red, showing
negative impact of curiosity traps.

(a)

(b)

Figure 9: Expanding T-Maze Results (see descriptions in figure 8).

## Discussion

As described in section 4, our experiments corroborated the hypotheses in the
introduction: namely, action entropy and curiosity improve agent exploration,
especially when both are implemented together as implied by the FEP;
prediction error curiosity can be negatively influenced by observational noise
also known as curiosity traps, while hidden state curiosity can be more
resilient to such curiosity traps. These results indicate that applying the
FEP can significantly benefit RL, encouraging agents to investigate and
comprehend causal structures which would otherwise be difficult or impossible
to understand. This could be beneficial for robots in dynamic environments, or
interactive systems automatically personalizing content delivery for its
users, or researchers seeking recommendations of directions to survey.
However, we have not yet attempted transferring behaviors learned in
simulation to physical agents, and only identified the optimized
hyperparameters through extensive brute-force testing.

The hidden state curiosity (and action entropy) foregrounded in this work
inherit from decompositions of expected free energy. Expected free energy in
this setting simply refers to a free energy functional of distributions over
(random) variables expected under a particular policy or path into the future.
The expected free energy could be regarded as a universal objective function
from the perspective of the physics of self organizing agents that have well-
defined or characteristic attracting sets. For a recent derivation of expected
free energy—from the perspective of statistical physics–please see Friston et
al., (2023). One interesting interpretation of expected free energy is in
terms of a dual aspect Bayes optimality. This follows from our decomposition
of expected free energy into expected information gain and expected extrinsic
reward. These are exactly the objective functions that underwrite the
principles of optimal Bayesian experimental design (Lindley,, 1956; MacKay,,
1992) and decision theory (Berger,, 2011), respectively. On this view,
intrinsic and extrinsic reward are two sides of the same coin, where
information and value have exactly the same currency (i.e., natural units).
The implication here is that one can think about rewards in terms of
information and, conversely, think about the value of information (Howard,,
1966) as intrinsically rewarding.

The expected information gain can be applied to any latent variables (i.e.
states or parameters) of a forward model. When applied to the latent states of
a generative model, the implicit intrinsic reward or motivation is sometimes
referred to as salience (Itti and Baldi,, 2009). Conversely, when applied to
the parameters of a forward model the expected information gain is sometimes
referred to as the novelty that underwrites curious behavior (Baldassarre et
al.,, 2014; Da Costa et al.,, 2020; Schmidhuber,, 2010; Schwartenbeck et al.,,
2019). This is important because we have absorbed the implicit active
inference and learning into a reinforcement learning scheme based upon the
actor-critic model. This kind of reinforcement learning identifies state-
action policies in the sense that an optimal policy is identified for every
given state. This means one cannot select policies that maximize information
gain about latent states (because each policy is conditioned upon being in a
particular state; as opposed to having posterior beliefs about latent states).
However, it is still possible to learn policies that, on average, are
information seeking; especially about the parameters of a generative model—as
in our case. The parameters in question here are the transition parameters of
the forward model. This leads to the interesting notion that one can learn
state-action policies that are information seeking, in exactly the way we have
demonstrated with the above numerical experiments. These are sometimes
referred to as epistemic habits (Friston et al.,, 2016): e.g., habitually
watching a certain news channel in the evening to seek information about what
happened during the day. This observation is potentially important because it
suggests there are lawful and learnable ways of foraging changing environments
for information; such as mazes that feature curiosity traps and change over
time.

Looking forward, there are multiple ways future research can explore hidden
state curiosity. For example, the PV-RNN architecture (Ahmadi and Tani,, 2019)
can introduce hierarchical processing within a VRNN (Chung et al.,, 2016).
Each layer of such a framework could generate hidden states with different
Multiple Timescale RNNs (MTRNNs) (Yamashita and Tani,, 2008; Jian et al.,,
2023), allowing agents to access both long-term and short-term memories. Using
this architecture, agents could have curiosity about the environment in
multiple temporal contexts.

Moreover, just as the hyperparameter $\alpha$ can be a parameter which
dynamically adjusts to satisfy target entropy $\bar{\mathcal{H}}$ (see section
2.1), it may be possible to adjust the hyperparameter $\eta$ dynamically to
satisfy target curiosity $\bar{\mathcal{C}}$. For example, like $\alpha$ is
optimized to minimize
$\log(\alpha)\cdot(\mathcal{H}(\pi(a_{t}|o_{t}))-\bar{\mathcal{H}})$, $\eta$
could be optimized to minimize
$\log(\eta)\cdot(D_{KL}[q(z_{t})||p(z_{t})]-\bar{\mathcal{C}})$. This may
refine the agent’s engagement with curiosity over time and help users select
optimal hyperparameters.

Also, future research should investigate how choice of $\eta$, $\beta$, and
the sizes of observations and inner states impact hidden state curiosity’s
reaction to curiosity traps. Although choice of $\eta$ and $\beta$ can be
considered configurable customization, it can be challenging to fine-tune
optimal pairings for ignoring useless noise in the task at hand111Generally,
in FEP-based schemes, a free hyperparameter can, in principle, be optimized
with respect to variational free energy. For simple hyperparameters, this is
usually best achieved with a line search over the hyperparameter to minimise
the path integral of variational free energy—as a bound on log marginal
likelihood or model evidence—accumulated over the time period in question..

Finally, the intrinsic reward for imitation shown by Kawahara et al., (2022)
in equation 2.3 should be investigated. This could enable agents to learn from
human demonstrations, particularly for rare or complex situations.

In conclusion, emulating behaviors associated with biological agents like
curiosity-driven exploration appears to be a promising frontier in advancing
AI. Although current RL agents can have great computational power, they cannot
yet achieve understandings as nuanced as inquisitive humans. The FEP offers a
principled way to bring such useful organic practices to artificial agents.
Our future hope is to apply hidden state curiosity to more intricate 3D agents
training to perform compositional action goals, delving into the FEP’s
influence on embodied cognition and the emergence of communication.

Find our code at github.com/oist-cnru/curious_maze.

### Acknowledgments

The authors are funded by OIST graduate school.

Thank you greatly to the anonymous reviewer.

## References

  * Ahmadi and Tani, (2019) Ahmadi, A. and Tani, J. (2019).  A novel predictive-coding-inspired variational rnn model for online prediction and recognition.  Neural Computation, 31:2025–2074.
  * Baldassarre et al., (2014) Baldassarre, G., Stafford, T., Mirolli, M., Redgrave, P., Ryan, R. M., and Barto, A. (2014).  Intrinsic motivations and open-ended development in animals, humans, and robots: an overview.  Frontiers in Psychology, 5.  Section: Cognitive Science.
  * Barto et al., (1983) Barto, A. G., Sutton, R. S., and Anderson, C. W. (1983).  Neuronlike adaptive elements that can solve difficult learning control problems.  IEEE Transactions on Systems, Man, and Cybernetics, SMC-13(5):834–846.
  * Berger, (2011) Berger, A. (2011).  Self-regulation: Brain, cognition, and development.  American Psychological Association.
  * Blundell et al., (2015) Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. (2015).  Weight uncertainty in neural network.  In International Conference on Machine Learning, pages 1613–1622. PMLR.
  * Chung et al., (2016) Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A., and Bengio, Y. (2016).  A recurrent latent variable model for sequential data.  arXiv preprint arXiv:1506.02216.  CIFAR Senior Fellow.
  * Da Costa et al., (2020) Da Costa, L., Parr, T., Sajid, N., Veselic, S., Neacsu, V., and Friston, K. (2020).  Active inference on discrete state-spaces: A synthesis.  Journal of Mathematical Psychology.  Review.
  * Friston et al., (2023) Friston, K., Da Costa, L., Sajid, N., Heins, C., Ueltzhöffer, K., Pavliotis, G. A., and Parr, T. (2023).  The free energy principle made simpler but not too simple.  Physics Reports, 1024:1–29.
  * Friston et al., (2016) Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., O’Doherty, J., and Pezzulo, G. (2016).  Active inference and learning.  Neuroscience & Biobehavioral Reviews, 68:862–879.
  * Haarnoja et al., (2018) Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018).  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  arXiv preprint arXiv:1801.01290.
  * Howard, (1966) Howard, R. (1966).  Decision analysis: Applied decision theory.
  * Itti and Baldi, (2009) Itti, L. and Baldi, P. (2009).  Bayesian surprise attracts human attention.  Vision Research, 49:1295–1306.
  * Jian et al., (2023) Jian, N. L., Zabiri, H., and Ramasamy, M. (2023).  Control of the multi-timescale process using multiple timescale recurrent neural network-based model predictive control.  Industrial and Engineering Chemistry Research, 62(15):6176–6195.
  * Kaplan and Friston, (2018) Kaplan, R. and Friston, K. J. (2018).  Planning and navigation as active inference.  Biological Cybernetics, 112(3):323–343.
  * Kawahara et al., (2022) Kawahara, D., Ozeki, S., and Mizuuchi, I. (2022).  A curiosity algorithm for robots based on the free energy principle.  In 2022 IEEE/SICE International Symposium on System Integration (SII), Narvik, Norway.
  * Kingma et al., (2015) Kingma, D. P., Salimans, T., and Welling, M. (2015).  Variational dropout and the local reparameterization trick.  In Advances in Neural Information Processing Systems, volume 28, pages 2575–2583.
  * Lindley, (1956) Lindley, D. V. (1956).  On a measure of the information provided by an experiment.  Annals of Mathematical Statistics, 27(4):986–1005.
  * MacKay, (1992) MacKay, D. J. C. (1992).  A practical bayesian framework for backpropagation networks.  Neural Computation, 4:448–472.
  * Millidge et al., (2020) Millidge, B., Tschantz, A., Seth, A. K., and Buckley, C. L. (2020).  On the relationship of active inference and control as inference.  Preprint.
  * Mnih et al., (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. (2015).  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533.
  * Oudeyer and Kaplan, (2007) Oudeyer, P.-Y. and Kaplan, F. (2007).  What is intrinsic motivation? a typology of computational approaches.  Frontiers in neurorobotics.  Reviewed by: Jeffrey L. Krichmar, The Neurosciences Institute, USA; Cornelius Weber, Johann Wolfgang Goethe University, Germany.
  * Parr and Friston, (2019) Parr, T. and Friston, K. J. (2019).  Generalised free energy and active inference.  Biological Cybernetics.
  * Pathak et al., (2017) Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. (2017).  Curiosity-driven exploration by self-supervised prediction.  In Proceedings of the 34th International Conference on Machine Learning, JMLR: W&CP, Sydney, Australia. JMLR.org.
  * Schmidhuber, (2010) Schmidhuber, J. (2010).  Formal theory of creativity, fun, and intrinsic motivation (1990-2010).  IEEE Transactions on Autonomous Mental Development, 2(3):230–247.
  * Schwartenbeck et al., (2019) Schwartenbeck, P., Passecker, J., Hauser, T. U., FitzGerald, T. H., Kronbichler, M., and Friston, K. J. (2019).  Computational mechanisms of curiosity and goal-directed exploration.  eLife, 8:e41703.
  * Tschantz et al., (2023) Tschantz, A., Baltieri, M., Seth, A. K., and Buckley, C. L. (2023).  Scaling active inference.  IEEE Xplore.
  * Tschantz et al., (2020) Tschantz, A., Millidge, B., Seth, A. K., and Buckley, C. L. (2020).  Reinforcement learning through active inference.  In Bridging AI and Cognitive Science (ICLR 2020 Workshop), Brighton, UK. University of Sussex and University of Edinburgh.
  * Watkins and Dayan, (1992) Watkins, C. J. and Dayan, P. (1992).  Q-learning.  Machine Learning, 8:279–292.
  * Yamashita and Tani, (2008) Yamashita, Y. and Tani, J. (2008).  Emergence of functional hierarchy in a multiple timescale neural network model: A humanoid robot experiment.  PLoS Comput Biol, 4(11):e1000220.
  * Yang and Nguyen, (2021) Yang, Z. and Nguyen, H. (2021).  Recurrent off-policy baselines for memory-based continuous control.  In Deep Reinforcement Learning Workshop, NeurIPS 2021.

Figure 10: Proportion of agents taking each exit in the T-maze. The correct
exit is colored light gray. Figure 11: Proportion of agents taking each exit
in the expanding T-maze. The correct exit is colored light gray.

## Appendix B

These tables illustrate details about models’ architectures in PyTorch. Each
table illustrates the parameters of a model layer by layer. PReLU refers to
LeakyReLU with leak-coefficient as a trainable parameter.

Forward |  |
---|---|---
Portion | Layer Type | Details
Image In | Convolution | channels in=4, channels out=16
|  |  kernel_size=(3, 3), stride=(1, 1),
|  |  padding=(1, 1), padding_mode=reflect
| PReLU | num_parameters=1
| Average Pooling | kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)
| Convolution | channels in=16, channels out=16,
|  |  kernel_size=(3, 3), stride=(1, 1),
|  |  padding=(1, 1), padding_mode=reflect
| PReLU | num_parameters=1
| Average Pooling | kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)
| Flatten |
| Linear | in_features=64, out_features=32, bias=True
| PReLU | num_parameters=1
Speed In | Linear | in_features=1, out_features=32, bias=True
| PReLU | num_parameters=1
Table 2: Architecture of the Forward Model (part 1). Forward (continued) |  |
---|---|---
Portion | Layer Type | Details
Action In | Linear | in_features=2, out_features=32, bias=True
| PReLU | num_parameters=1
$\mu^{p}$ ($z^{p}$ Mean ) | Linear | in_features=64, out_features=32, bias=True
| PReLU | num_parameters=1
| Linear | in_features=32, out_features=32, bias=True
| Tanh |
$\sigma^{p}$ ($z^{p}$ STD) | Linear | in_features=64, out_features=32, bias=True
| PReLU | num_parameters=1
| Linear | in_features=32, out_features=32, bias=True
| Softplus | beta=1, threshold=20
$\mu^{q}$ ($z^{q}$ Mean) | Linear | in_features=128, out_features=32, bias=True
| PReLU | num_parameters=1
| Linear | in_features=32, out_features=32, bias=True
| Tanh |
$\sigma^{q}$ ($z^{q}$ STD) | Linear | in_features=128, out_features=32, bias=True
| PReLU | num_parameters=1
| Linear | in_features=32, out_features=32, bias=True
| Softplus | beta=1, threshold=20
Table 3: Architecture of the Forward Model (part 2). Forward (continued) |  |
---|---|---
Portion | Layer Type | Details
GRU | Gated RNN | input size=32, hidden size=32
Image Out | Linear | in_features=64, out_features=16, bias=True
| PReLU | num_parameters=1
| Reshape |
| Convolution | channels in=4, channels out=16,
|  |  kernel_size=(3, 3), stride=(1, 1),
|  |  padding=(1, 1), padding_mode=reflect
| PReLU | num_parameters=1
| Upsampling | scale_factor=2, mode=‘bilinear’
| Convolution | channels in=16, channels out=16,
|  |  kernel_size=(3, 3), stride=(1, 1),
|  |  padding=(1, 1), padding_mode=reflect
| PReLU | num_parameters=1
| Upsampling | scale_factor=2, mode=‘bilinear’
| Convolution | channels in=16, channels out=16,
|  |  kernel_size=(3, 3), stride=(1, 1),
|  |  padding=(1, 1), padding_mode=reflect
| PReLU | num_parameters=1
| Convolution | channels in=16, channels out=4,
|  |  kernel_size=(1, 1), stride=(1, 1)
Table 4: Architecture of the Forward Model (part 3). Forward (continued) |  |
---|---|---
Portion | Layer Type | Details
Speed Out | Linear | in_features=64, out_features=32, bias=True
| PReLU | num_parameters=1
| Linear | in_features=32, out_features=32, bias=True
| PReLU | num_parameters=1
| Linear | in_features=32, out_features=1, bias=True
Table 5: Architecture of the Forward Model (part 4). Actor |  |
---|---|---
Portion | Layer Type | Details
h In | Linear | in_features=32, out_features=32, bias=True
| PReLU | num_parameters=1
| Linear | in_features=32, out_features=32, bias=True
| PReLU | num_parameters=1
| Linear | in_features=32, out_features=32, bias=True
| PReLU | num_parameters=1
| Linear | in_features=32, out_features=32, bias=True
| PReLU | num_parameters=1
$\mu$ | Linear | in_features=32, out_features=2, bias=True
$\sigma$ | Linear | in_features=32, out_features=2, bias=True
| Softplus | beta=1, threshold=20
Table 6: Architecture of the Actor Model. The final action is $a=\tanh(x\sim\mathcal{N}(\mu,\sigma))$, and the log probability of that action is $\log(\mathcal{N}(x|\mu,\sigma^{2}))-\log(1-a^{2}+10^{-6})$. Critic |  |
---|---|---
Portion | Layer Type | Details
Image In | Convolution | channels in=4, channels out=16,
|  |  kernel_size=(3, 3), stride=(1, 1),
|  |  padding=(1, 1), padding_mode=reflect
| PReLU | num_parameters=1
| Average Pooling | kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)
| Convolution | channels in=16, channels out=16,
|  |  kernel_size=(3, 3), stride=(1, 1),
|  |  padding=(1, 1), padding_mode=reflect
| PReLU | num_parameters=1
| Average Pooling | kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)
| Flatten |
| Linear | in_features=64, out_features=32, bias=True
| PReLU | num_parameters=1
Speed In | Linear | in_features=1, out_features=32, bias=True
| PReLU | num_parameters=1
Action In | Linear | in_features=2, out_features=32, bias=True
| PReLU | num_parameters=1
Table 7: Architecture of the Critic Model (part 1). Not pictured: concatenation of Image In, Speed In, and Action In for GRU input. Critic (continued) |  |
---|---|---
Portion | Layer Type | Details
GRU | Gated RNN | input size=96, hidden size=32
Q Out | Linear | in_features=32, out_features=32, bias=True
| PReLU | num_parameters=1
| Linear | in_features=32, out_features=1, bias=True
Table 8: Architecture of the Critic Model (part 2).