# Ballistic thermal rectification

Lifa Zhang  Department of Physics and Centre for Computational Science and
Engineering, National University of Singapore, Singapore 117542, Republic of
Singapore  Jian-Sheng Wang  Department of Physics and Centre for Computational
Science and Engineering, National University of Singapore, Singapore 117542,
Republic of Singapore  Baowen Li  Electronic address<EMAIL_ADDRESS>Department of Physics and Centre for Computational Science and Engineering,
National University of Singapore, Singapore 117542, Republic of Singapore  NUS
Graduate School for Integrative Sciences and Engineering, Singapore 117456,
Republic of Singapore

(8 Feb 2010)

###### Abstract

We study ballistic thermal transport in three-terminal atomic nano-junctions
by the nonequilibrium Green’s function method. We find that there is ballistic
thermal rectification in asymmetric three-terminal structures because of the
incoherent phonon scattering from the control terminal. With spin-phonon
interaction, we also find the ballistic thermal rectification even in
symmetric three-terminal paramagnetic structures.

###### pacs:

05.60.-k, 44.10.+i, 66.70.-f

_Introduction_ Phononics, the study of information processing and controlling
of heat flow by phonons, is an emerging new field that is attracting
increasing attention Wang2008 . Specifically, researchers have recently
modeled and built thermal rectifiers rectifiers , thermal transistors
transistor , thermal logical gates logicgate , and thermal memory memory ,
which are the basic components of functional thermal devices. The most
fundamental phononic component is the thermal rectifier – a device that allows
larger conduction in one direction than in the opposite direction when it is
driven far enough from equilibrium. The effect of thermal rectification is
well known to be realized by combining the system inherent anharmonicity with
structural asymmetry rectifiers ; transistor ; logicgate ; memory ; segal2005
; wu2009 ; ruokola2009 ; zhang20091 . Whether the rectification can happen in
harmonic systems, that is, ballistic rectification, is still unknown, although
one recent paper hopkins2009  discussed it, in which the authors get different
heat conductances by exchanging the heat baths with different structures
connected to an asymmetric center part, thus the different conductances come
from the totally different systems. As we know, the thermal transport in
nanoscale materials, which is very promising for thermal devices, can be
regarded as ballistic because of their small sizes in comparison with the
phonon mean free path. Therefore, it is highly desirable to investigate
whether the ballistic thermal rectification can be realized in harmonic
systems, and to explore the necessary conditions for thermal rectification.

The ballistic thermal transport in two-terminal junctions can be described by
the Landauer formula. Since the temperatures enter only through the Bose
distribution, it is obvious that if we reverse the heat bath temperatures, the
heat flux only changes sign, and no rectification is expected. How about the
ballistic thermal transport in multiple-terminal junctions? The theory for
multiple-terminal electric transport was proposed as the Landauer-Büttiker
conductance formula datta1995 ; buttiker ; blanter2000 , and was applied to
thermal transport recently sun2002 ; ming2007 ; zhang20092 . From the
electronic transport in three-terminal system blanter2000 , we know that the
third terminal can introduce incoherence or phase breaking to the transport.
So it is our interest to investigate whether a multiple-terminal junction is a
proper option for ballistic thermal transport, that is, whether the
incoherence through the third terminal can induce rectification effect. We
will take the nonequilibrium Green’s function (NEGF) approach negf ; refnegf ;
ruokola2009 ; ojanen2008 . NEGF is widely applied to electronic and thermal
transport, and is successful to study the spin Hall effect and phonon Hall
effect in junctions Sheng2005 ; zhang20092 .

Figure 1: (Color online) The three-terminal junction setup to study the
ballistic thermal transport. The left and right leads have temperatures
$T_{L}$ and $T_{R}$, the control terminal lead is adjusted to be $T_{C}$ or
$T^{{}^{\prime}}_{C}$ so that the heat flux from this lead is zero in the
forward ($T_{L}=T_{+}$, $T_{R}=T_{-}$) or backward ($T_{L}=T_{-}$,
$T_{R}=T_{+}$) process. $T_{+}$ and $T_{-}$ are the temperatures of the hot
and cold baths, respectively.

_Model and Method_ We consider the ballistic thermal transport in a three-
terminal nano-junction as shown in Fig. 1, where a two-dimensional lattice
sample, which is a honeycomb lattice, is connected with three ideal semi-
infinite leads. The masses are coupled through nearest neighbors by elastic
springs (with longitudinal and transverse force constants). We denote the
center lattice as $N_{R}\times N_{C}$, $N_{R},\,N_{C}$ correspond to the
number of rows and columns, respectively. The external magnetic field can be
perpendicularly applied to this part. We use $N_{CL}$ to denote the number of
columns of the control lead and $N_{CD}$ to denote the number of columns
deviate from the middle of the center part; if $N_{CD}=0$, the whole setup is
symmetric. In Fig. 1, $N_{R}=9,\,N_{C}=8,\,N_{CL}=4,\,N_{CD}=-2$. The
Landauer-Büttiker formula can be expressed as,

$J_{\alpha}=\sum\limits_{\beta\neq\alpha}{\int_{0}^{\infty}{\frac{{d\omega}}{{2\pi}}}\hbar\omega\,\tau_{\beta\alpha}(\omega)\bigl{[}n(T_{\alpha})-n(T_{\beta})\bigr{]}}.$
(1)

Here, $\tau_{\beta\alpha}$ is the transmission coefficient from the $\alpha$th
bath to the $\beta$th bath; and
$n(T_{\alpha})=(e^{\hbar\omega/k_{B}T_{\alpha}}-1)^{-1}$ is the Bose
distribution with $T_{\alpha}$ being the temperature of the $\alpha$th heat
bath. We set $\hbar=1$ and $k_{B}=1$ in the following calculation. Therefore,
in the forward process, $T_{L}=T_{+}$, $T_{R}=T_{-}$, we obtain

$J_{+}\\!\\!=\\!\\!\int\\!{\frac{{\omega
d\omega}}{{2\pi}}\bigl{\\{}\tau_{RL}(\omega)[n(T_{+})-n(T_{-})]}+\tau_{CL}(\omega)[n(T_{+})-n(T_{C})]\bigr{\\}}$
(2) $J_{C}\\!\\!=\\!\\!\int\\!{\frac{{\omega
d\omega}}{{2\pi}}\bigl{\\{}\tau_{LC}(\omega)[n(T_{C})-n(T_{+})]}+\tau_{RC}(\omega)[n(T_{C})-n(T_{-})]\bigr{\\}}$
(3)

Similarly we can obtain the heat fluxes $J_{-}$ and $J^{{}^{\prime}}_{C}$ in
the backward process $T_{L}=T_{-}$, $T_{R}=T_{+}$. From the equations of
$J_{C}=0$ and $J^{{}^{\prime}}_{C}=0$ we can obtain the temperatures of the
control bath $T_{C}$ and $T^{{}^{\prime}}_{C}$; inserting them to the formulae
of $J_{+}$ and $J_{-}$, by the definition of rectification as

$R=(J_{+}-J_{-})/{\rm max}\\{J_{+},J_{-}\\},$ (4)

we can calculate the rectification of this model. If the system is in the
linear response regime or in the classic limit, the heat flux from the
$\alpha$th lead can be expressed as
$J_{\alpha}=\sum\limits_{\beta\neq\alpha}\sigma_{\beta\alpha}(T_{\alpha}-T_{\beta})$,
we set
$T_{+}=T_{0}+\Delta,\,T_{-}=T_{0}-\Delta,\,T_{C}=T_{0}+\delta,\,T^{{}^{\prime}}_{C}=T_{0}+\delta^{{}^{\prime}}$,
then we obtain

$\delta=-\delta^{\prime}=\frac{{\sigma_{LC}-\sigma_{RC}}}{{\sigma_{LC}+\sigma_{RC}}}\Delta;$
(5)
$J_{+}=J_{-}=2\Delta\left(\sigma_{RL}+\frac{{\sigma_{CL}\sigma_{RC}}}{{\sigma_{LC}+\sigma_{RC}}}\right).$
(6)

So there is no rectification in the linear response regime or in the classic
limit. In order to get thermal rectification, we should consider the quantum
regime out of linear response, and the key work is to compute the transmission
coefficients among the heat baths.

In this paper, in addition to the structural asymmetry, we also can introduce
the same spin-phonon interaction as in Refs. Sheng2006 ; Kagan2008 ; wang2009
; zhang20092  in order to break a time-reversal symmetry. The Hamiltonian of
our model can be written as

$H=\sum\limits_{\alpha=0,L,R,C}H_{\alpha}+\sum\limits_{\beta=L,R,C}{U_{\beta}^{T}V_{\beta,0}U_{0}}+U_{0}^{T}AP_{0},$
(7)

where
$H_{\alpha}=\frac{1}{2}\left(P_{\alpha}^{T}P_{\alpha}+U_{\alpha}^{T}K_{\alpha}U_{\alpha}\right)$.
The superscript $T$ denotes matrix transpose. Here, $0,L,R,C$ correspond to
the center region, left, right, and control leads, respectively. $U_{\alpha}$
is a column vector for mass reduced displacements in region $\alpha$,
$P_{\alpha}$ is the associated conjugate momentum vector, and $K_{\alpha}$ is
the force constant matrix, $V_{\beta,0}=(V_{0,\beta})^{T}$ is the coupling
matrix between the $\beta$th lead and the central region. $A$ is a block
diagonal matrix with diagonal elements $\left({\begin{array}[]{*{20}c}0&h\\\
-h&0\\\ \end{array}}\right).$ $h$ is a model parameter which is supposed to be
proportional to the magnetic field. In the first part of this paper, we set
$h=0$; it is a standard harmonic phononic system.

Figure 2:  (Color online) Rectification as a function of relative temperature
difference of the hot and cold heat baths. The parameter of the setup is
$N_{R}=9,\,N_{C}=16,\,N_{CL}=2$. The temperature of the heat bath are
$T_{+}=T_{0}(1+\Delta)$ and $T_{-}=T_{0}(1-\Delta)$, where $T_{0}=0.2$ is the
mean temperature. The solid square, solid circle, solid triangle, diamond,
hollow triangle, hollow circle, hollow square correspond to $N_{CD}=-7$, -5,
-3, 0, 3, 5 and 7, respectively.

The retarded Green’s function for the central region in frequency domain is
zhang20092

$G^{r}[\omega]=\Bigl{[}(\omega+i\eta)^{2}-K_{0}-\Sigma^{r}[\omega]-A^{2}+2i\omega
A\Bigr{]}^{-1}.$ (8)

Here, $\Sigma^{r}=\sum\limits_{\beta=L,C,R}{\Sigma_{\beta}^{r}}$, and
$\Sigma_{\beta}^{r}=V_{0,\beta}g_{\beta}^{r}V_{\beta,0}$ is the self-energy
due to interaction with the heat baths,
$g_{\beta}^{r}=[(\omega+i\eta)^{2}-K_{\beta}]^{-1}$. $\eta$ is an
infinitesimal real positive quantity. The surface Green’s functions of the
leads $g_{\beta}^{r}$ are obtained following the algorithms of Ref. refnegf .
The transmission coefficient is

$\tau_{\beta\alpha}[\omega]={\rm{Tr}}(G^{r}\Gamma_{\beta}G^{a}\Gamma_{\alpha}),$
(9)

where,
$\Gamma_{\alpha}=i\bigl{(}\Sigma_{\alpha}^{r}[\omega]-\Sigma_{\alpha}^{a}[\omega]\bigr{)}$,
$G^{a}=(G^{r})^{\dagger}$, and
$\Sigma_{\alpha}^{a}=(\Sigma_{\alpha}^{r})^{\dagger}$.

_Results and Discussions_ Firstly, we consider the ballistic thermal transport
in an asymmetric structure without an external magnetic field. In the
following simulation, we set the longitudinal spring constant $k_{L}=1.0$, and
the transverse one $k_{T}=0.25$. If the control lead is connected to the
middle of upper edge of the center, that is, $N_{CD}=0$, the forward process
and backward one are exactly the same; no rectification will be expected, as
shown in Fig. 2 (the diamond symbols). We will obtain the same result of no
rectification if we replace the center honeycomb lattice with the square
lattice same as the leads, when the whole system is symmetric wherever we put
the control lead. If the control lead moves away from the center, the
rectification effect appears. When the lead is moved the same distance to the
left or right, the rectification coefficient has the same magnitude but
opposite sign, which is because that the two cases only exchange the value of
$J_{+}$ and $J_{-}$. If the distance between the control lead and the middle
of the center part is longer, the rectification effect is larger. In Fig. 2,
we can see that the case of $N_{CD}=\pm 7$, when the control lead is next to
left or right lead, has biggest rectification. The rectification increases
with the temperature difference at far-from-linear-response regime.

Figure 3:  (Color online) The difference of transmission coefficients:
$\tau_{LC}-\tau_{RC}$, as a function of frequency. The parameter of the setup
is $N_{R}=9,\,N_{C}=16,\,N_{CL}=2$. The solid, dot curves correspond to
$N_{CD}=-3$ and $N_{CD}=-7$, respectively. .

From the above formulae, we find if the transmissions $\tau_{LC}$ and
$\tau_{RC}$ are linearly dependent (here
$\tau_{\alpha\beta}=\tau_{\beta\alpha}$ because of time reversal symmetry),
the rectification will be zero. Figure 3 shows that the transmission
coefficients $\tau_{LC}$ and $\tau_{RC}$ are not linearly dependent, so that
the rectification is nonzero. In some frequency domains,
$\tau_{LC}>\tau_{RC}$; and in other frequency domains, $\tau_{LC}<\tau_{RC}$.
With the distance increasing, that is, from $N_{CD}=-3$ to $N_{CD}=-7$, the
difference of $\tau_{LC}$ and $\tau_{RC}$ enlarges in most part of the whole
frequency domain; so that the rectification coefficient increases. Because of
scattering from the third bath – the control lead, the total thermal transport
from the left lead to the right one is partially incoherent. This phonon
incoherence induces rectification. In the whole system the Hamiltonian is
quadratic, that is, there is no nonlinearity or anharmonicity, but we still
can obtain rectification. Therefore, the phonon incoherence, which can be
induced by either nonlinearity or scattering lead, is the necessary condition
for thermal rectification.

Figure 4:  (Color online) (a) Thermal rectification as function of relative
temperature difference $\Delta$ for different width of control lead, at
$T_{0}=0.2$; (b) Thermal rectification as function of mean temperature for
different relative temperature difference. $N_{CL}=2,\,N_{CD}=-7$. For both
(a) and (b), $N_{R}=9,N_{C}=16$, $T_{+}=T_{0}(1+\Delta)$ and
$T_{-}=T_{0}(1-\Delta)$.

The control lead acts as a scattering source, which makes the phonon transport
incoherent, so that the rectification comes out. However, the width of the
control lead does not quite affect the whole thermal transport, which is shown
in Fig. 4(a). We make the control lead next to the left lead, and find that
the rectification changes little when we increase the width of the lead. If
the width of control lead increases further, the rectification decreases
because the asymmetry decreases. Figure 4(b) shows the rectification
dependence on temperature, and reproduce the reversal of rectification found
in Ref. zhang20091 . At a low temperature, the contribution to thermal
transport only comes from the low frequency phonons; if the temperature
increases, more high frequency phonons will contribute to the heat transport.
From Fig. 3, the relations between transmissions $\tau_{LC}$ and $\tau_{RC}$
in low frequency domain and high frequency domain are opposite, so that the
rectification reverses with the temperature increasing. When the temperature
increases further, the system will go to the classic limit, the rectification
disappears.

Figure 5:  (Color online) (a) Thermal rectification as function of relative
temperature difference $\Delta$ for different external magnetic fields. (b)
Thermal rectification as a function of magnetic field $h$. For both (a) and
(b): $N_{R}=9,\,N_{C}=16,\,N_{CL}=2,\,N_{CD}=0$. $T_{+}=T_{0}(1+\Delta)$,
$T_{-}=T_{0}(1-\Delta)$. $T_{0}$=0.2.

From the previous work rectifiers ; segal2005 ; zeng08  on thermal
rectification, we know that in order to get rectification, we need the
structural asymmetry. However, in the nanoscale rectifier, it is not easy to
control the structural asymmetry or not easily distinguish the rectification
direction by the structural asymmetry. Is there any other means to introduce
asymmetry to induce rectification? From the study of phonon Hall effect
Sheng2006 ; Kagan2008 ; wang2009 ; zhang20092 , it is known that the magnetic
field can influence the thermal transport by the spin-phonon interaction. Thus
the magnetic field can break the symmetry of the phonon transport. We apply an
external magnetic field perpendicular to the center part of a symmetric
structure to study the ballistic thermal transport, the results are shown in
Fig. 5. The thermal rectification effect as a function of the temperature
difference is shown in Fig. 5(a). $R$ increases with the temperature
difference, and can be about $3\%$ if $\Delta=0.8$ and $h=0.3$ at $T_{0}=0.2$.
Figure 5(b) shows that the rectification can monotonically increase with the
external magnetic field in the range of $h=0\sim 0.3$. The applied magnetic
field breaks the symmetry of the phonon transport system through the spin-
phonon interaction, so the transmission coefficient from the control lead to
left one is different from that to right one. Figure 6 shows that the two
transmission coefficients are not linearly dependent; and with the increasing
of magnetic field, the difference of these two transmissions will enlarge,
which induce bigger rectification effect. Here all quantities are
dimensionless, if we use the parameters for real materials, the rectification
coefficient may be different, but would be likely in the range of few per
cents.

Although all our values for rectification are small, if we can devise a chain
of such ballistic rectifiers, the total rectification can be as large as we
need. Because for most nanoscale materials the thermal transport is ballistic
and temperature can be applied far from linear response regime, our prediction
can be tested by experiments. The asymmetric quasi-two-dimensional nano-
structure of any material can have ballistic thermal rectification, such as a
graphene sheet. Or the symmetric structure with spin-obit interaction, such as
paramagnetic dielectrics, can also have ballistic rectification applied an
perpendicular magnetic field.

Figure 6:  (Color online) The difference of transmission coefficients:
$\tau_{LC}-\tau_{RC}$, as a function of frequency for different applied
magnetic fields. The parameter of the setup is
$N_{R}=9,\,N_{C}=16,\,N_{CL}=2,\,N_{CD}=0$. The solid, dot curves correspond
to $h=0.1$ and $h=0.3$, respectively.

_Conclusions_ Using the nonequilibrium Green’s function method, we have
studied ballistic thermal transport in three-terminal atomic nano-junctions.
Adjusting the temperature of the control lead in order to make the heat flux
from this lead be zero, we can calculate the thermal rectification effect. In
the quantum regime out of linear response, there is ballistic thermal
rectification in asymmetric three-terminal structures because of incoherent
phonon scattering from the control terminal. Through spin-phonon interaction,
we also find the ballistic thermal rectification in symmetric three-terminal
paramagnetic structures applied an external magnetic field. Therefore, not the
nonlinearity, but the phonon incoherence, which can be induced by nonlinearity
or scattering boundary or scattering lead, is the necessary condition for
thermal rectification. Another necessary condition is asymmetry, not
necessarily being structural asymmetry, which can be introduced by an applied
external magnetic field through the spin-phonon interaction.

_Acknowledgements_ We thank Xiaoxi Ni, Jie Ren and Jie Chen for fruitful
discussions. LZ and BL are supported by the grant R-144-000-203-112 from
Ministry of Education of Republic of Singapore. JSW acknowledges support from
a faculty research grant R-144-000-257-112 of NUS.

## References

  * (1) L. Wang and B. Li, Physics World 21, No.3, 27 (2008).
  * (2) B. Li, L. Wang, and G. Casati, Phys. Rev. Lett. 93, 184301 (2004); B Li, J.-H Lan, L Wang, ibid 95, 104302 (2005); C. W. Chang, D. Okawa, A. Majumdar, and A. Zettl, Science 314, 1121 (2006). J.-H Lan and B Li, Phys. Rev. B 74, 214305 (2006).
  * (3) B. Li, L. Wang, and G. Casati, Appl. Phys. Lett. 88, 143501 (2006); W. C. Lo, L. Wang, and B. Li, J. Phys. Soc. Jpn. 77, 054402 (2008).
  * (4) L. Wang and B. Li, Phys. Rev. Lett. 99, 177208 (2007).
  * (5) L. Wang and B. Li, Phys. Rev. Lett. 101, 267203 (2008).
  * (6) D. Segal and A. Nitzan, Phys. Rev. Lett. 94, 034301 (2005).
  * (7) L.-A. Wu and D. Segal, Phys. Rev. Lett. 102, 095503 (2009).
  * (8) T. Ruokola, T. Ojanen, and A. -P. Jauho, Phys. Rev. B 79, 144306 (2009).
  * (9) L. Zhang, Y-H Yan, C.-Q Wu, J. -S. Wang, and B. Li, Phys. Rev. B 80, 172301 (2009).
  * (10) P. E. Hopkins and J. R. Serrano, Phys. Rev. B 80, 201408(R) (2009).
  * (11) S. Datta, Electronic Transport in Mesoscopic Systems (Cambridge Univ. Press, 1995).
  * (12) M. Büttiker, Phys. Rev. Lett. 57, 1761 (1986); M. Büttiker, IBM J. Res. Developm. 32, 317 (1988); Phys. Rev. B. 38, 9375(1988).
  * (13) Y. M. Blanter and M. Büttiker, Phys. Rep. 336, 1 (2000).
  * (14) Q. -F. Sun, P. Yang, and H. Guo, Phys. Rev. Lett. 89, 175901 (2002).
  * (15) Y. Ming, Z. X. Wang, Q. Li, and Z. J. Ding, Appl. Phys. Lett. 91, 143508 (2007).
  * (16) L. Zhang, J. -S. Wang and B. Li, New J. Phys. 11, 113038 (2009).
  * (17) H. Haug and A. P. Jauho, Quantum Kinetics in Transport and Optics of Semiconductors (Springer, 1996).
  * (18) J.-S. Wang, J. Wang, and N. Zeng, Phys. Rev. B 74, 033408 (2006); J.-S. Wang, J. Wang, and J. T. Lü, Eur. Phys. J. B 62, 381 (2008); L. Zhang, J. -S. Wang and B. Li, Phys. Rev. B 78, 144416 (2008).
  * (19) T. Ojanen and A. -P. Jauho, Phys. Rev. Lett. 100, 155902 (2008)
  * (20) L. Sheng, D. N. Sheng, and C. S. Ting, Phys. Rev. Lett. 94, 016602 (2005).
  * (21) N. Zeng and J.-S. Wang, Phys. Rev. B 78, 024305 (2008).
  * (22) L. Sheng, D. N. Sheng, and C. S. Ting, Phys. Rev. Lett. 96, 155901 (2006).
  * (23) Yu. Kagan and L. A. Maksimov, Phys. Rev. Lett. 100, 145902 (2008).
  * (24) J.-S. Wang and L. Zhang, Phys. Rev. B 80, 012301 (2009).


# Real-Time Recurrent Reinforcement Learning

Julian Lemmel, Radu Grosu

###### Abstract

Recent advances in reinforcement learning, for partially-observable Markov
decision processes (POMDPs), rely on the biologically implausible
backpropagation through time algorithm (BPTT) to perform gradient-descent
optimisation. In this paper we propose a novel reinforcement learning
algorithm that makes use of random feedback local online learning (RFLO), a
biologically plausible approximation of real-time recurrent learning (RTRL) to
compute the gradients of the parameters of a recurrent neural network in an
online manner. By combining it with TD($\lambda$), a variant of temporal-
difference reinforcement learning with eligibility traces, we create a
biologically plausible, recurrent actor-critic algorithm, capable of solving
discrete and continuous control tasks in POMDPs. We compare BPTT, RTRL and
RFLO as well as different network architectures, and find that RFLO can
perform just as well as RTRL while exceeding even BPTT in terms of complexity.
The proposed method, called real-time recurrent reinforcement learning
(RTRRL), serves as a model of learning in biological neural networks mimicking
reward pathways in the mammalian brain.

## Introduction

Model-free deep reinforcement-learning algorithms (DRLs) leveraging recurrent
neural networks (RNNs), have recently been shown to serve as a strong baseline
for a wide range of partially-observable Markov decision processes (POMDPs)
(Ni, Eysenbach, and Salakhutdinov 2022). Gradient-based DRLs, such as
temporal-difference (TD), have been shown to be sample efficient, and come
with provable convergence guarantees when using linear value-functions.

RNNs and DRLs were originally inspired by biological neural networks, which
are in general recurrent, and subject to synaptic plasticity, mediated by a
low-dimensional reward signal (Wise 2004). However, the indispensable
backpropagation through time algorithm (BPTT) they use (Werbos 1990), appears
to be biologically implausible, due to the need for both distinct forward and
backward phases, and weight transport (Bartunov et al. 2018). Thus, with BPTT,
RNN-DRLs renounce any biological interpretation.

The question we tried to answer in this paper was whether biological versions
of real-time recurrent learning (RTRL) (Williams and Zipser 1989), like
random-feedback online learning (RFLO) (Murray 2019), would be able to learn
and solve POMDP tasks, too. We positively answer this question by showing that
TD($\lambda$), the most basic version of a recurrent actor-critic algorithm
with eligibility traces and no other enhancements, such as the batched
experience replay, is able to solve both discrete and continuous POMDP tasks
in a biologically-plausible fashion, and in many cases converges much faster
than BPTT. We call our new method RTRRL (real-time recurrent reinforcement
learning). RTRRL shows superior exploration abilities when compared with state
of the art, and exceeds BPTT in terms of time and space complexity when used
with RFLO.

Figure 1: RTRRL implements Meta-RL as defined in Wang et al. (2018) by feeding
in the past action $a_{t-1}$ and reward $r_{t}$ to the CT-RNN alongside the
observation $o_{t}$. The latent state $h_{t}$ computed by the CT-RNN is used
to produce the next action $a_{t}$ and the value estimate $\hat{v}_{t}$
through a linear mapping.

The two main objections for BPTT-plausibility in biological neural networks
are the necessity for: (1)  _Shared weights_ , for forward and backward
connections and (2)  _Reciprocal error-transport_ , meaning that there is a
form of propagating back the errors without interfering with neural activity.
(Bartunov et al. 2018) We rephrase the premises for biologically plausible
learning similarly to (Bartunov et al. 2018). A biologically plausible
learning algorithm has to be: (1)  _Local_ , up to some low-dimensional reward
signal, (2)  _Online_ , in the sense that the updates are computed during the
feedforward pass, and (3)  _Without weight transport_ , in its error
computation. We elaborate on these requirements below.

Fully online. The computation of updates should not depend on alternating
forward and backward phases. Conceivably, such phases could be implemented
using pacemaker neurons. However, there must not be any freezing of values,
which occurs in BPTT. Furthermore, parallel streams of experience, such as in
batched environments of modern DRLs, violate this constraint, as a biological
agent can only interact with the singular environment in which it is situated.

No weight transport. Synapses that propagate back-error signals cannot have
their strength mirroring the strength of forward synapses. This is heavily
violated by backpropagation since backward pathways need access to and utilize
parameters that were used in the forward pass.

No horizontal gradient communication. Individual neurons $x_{k}$ are unlikely
to communicate their activation gradients $\nabla_{w_{ij}}x_{k}(t)$, with
respect to the synaptic parameters $w_{ij}$, that are not in their vicinity.
In other words, intermediate (source) neurons should not be able to tell other
(target) neurons how each of the synaptic weights of the source influence the
dynamics of the target, nor vice-versa.

Previous work on feedback alignment (Lillicrap et al. 2014) demonstrated that
weight transport is not strictly necessary for training deep neural networks.
They showed that randomly initialized feedback matrices, used for propagating
back gradients to previous layers, lead to acceptable function approximators,
and that the forward weights will align with the fixed backward weights during
training.

Online supervised learning for RNNs was shown to be possible by using RTRL
(Williams and Zipser 1989). More recently, a range of more computationally
efficient variants were introduced. (Tallec and Ollivier 2017; Roth,
Kanitscheider, and Fiete 2018; Mujika, Meier, and Steger 2018; Murray 2019).
Marschall, Cho, and Savin (2020) proposed a unifying framework for all these
variants and assessed their biological plausibility while showing that many of
them can perform on-par with BPTT. RFLO (Murray 2019), stands out due to its
biological update rule, that still performed well in the experiments of
Marschall, Cho, and Savin (2020).

Our work shows that online reinforcement learning for RNN-DRLs is possible
with RTRRL-RFLO, which fulfills all our premises for biologically plausible
learning. Our algorithm combines online RNN gradient computation with
TD($\lambda$). We create a fully online learning neural controller that does
not require multi-step unrolling, weight transport or communicating gradients
over long distances. Our algorithm succeeds in learning a policy from one
continuing stream of experiences alone, meaning that no large batches are
needed for reducing variance of the gradients. To the best of our knowledge,
the proposed RTRRL-RFLO, is the first fully biologically plausible recurrent
RL algorithm.

## Related Work

Neuroscience has turned to reinforcement learning (RL) in order to explain
learning in biology and expanding our understanding of the workings of reward
signaling in the brains of multi-cellular organisms. Recent work was able to
come up with RL algorithms inspired by findings in neuroscience that exhibit
activation patterns also found in the brains of mammals. Wang et al. (2018)
showed that RNNs can be trained to solve a general family of parameterized
tasks, where all instances follow a common structure, and that the activations
in trained RNNs mimic dopaminergic reward prediction errors (RPEs) measured in
primates. They show that the network is capable of inferring the underlying
parameters of each new task and subsequently solve unseen instances after
training. In particular, they use LSTM cells that receive the observation,
past action, and reward as inputs, and compute a latent state from which the
next action and value estimate are inferred by a linear mapping. Figure 1
shows the model architecture, which is also used in RTRRL.

This approach is called Meta-RL, since the RNN effectively learns to implement
an _inner_ RL algorithm that empowers it to learn about, and solve tasks on
its own. Behrens et al. (2018) argue that the timescale of plasticity
expressed through dopaminergic synapses in the prefrontal cortex, is
magnitudes below the time needed for primates to solve tasks, hinting at a
hierarchically layered RL algorithms reminiscent of the framework of Wang et
al. (2018).

Johard and Ruffaldi (2014) used two separate eligibility traces to approximate
the gradient of the discounted reward in a biologically plausible
connectionist actor-critic setting. They used a modified cascade-correlation
algorithm with eligibility traces for training, meaning that the initial small
neural network grows sequentially. In the paper, the show that the method
outperforms regular policy-gradient methods on the CartPole environment.

In his Master’s Thesis, Chung (2019) introduces a network architecture similar
to RTRRL that consists of a recurrent backbone and linear TD heads.
Convergence for the RL algorithm is proven assuming the learning rate of the
RNN being magnitudes below those of the heads. Albeit the similarity in
network structure, gradients were nonetheless computed using biologically
implausible BPTT.

Ladosz et al. (2022) extended DQNs by adding a neural network head called
MOHN, that is subject to Hebbian learning using eligibility traces. Their
algorithm significantly outperforms vanilla DQN on confounding POMDPs while
slwoing down execution time by 66%.

Another approach to recurrent RL was proposed by Javed et al. (2023) that
employs an efficient variant of RTRL. Similar to the cascade-correlation used
in (Johard and Ruffaldi 2014), they train a recurrent neural network
constructively i.e. one neuron at a time, keeping preceding neuron’s
parameters fixed. Although utilizing RTRL, their algorithms complexity is
reduced to be on par with BPTT.

Recently, Ororbia and Mali (2022) introduced a biologically plausible model-
based RL framework called active predicitve coding that enables online
learning of tasks with extremely sparse feedback. Their algorithm combines
predictive coding and active inference, two concepts grounded in neuroscience.
Neural network parameters are trained using Hebbian-like update rules that are
combined with component-wise surrogate losses.

## Background

#### Temporal-Difference Learning (TD).

This method combines ideas from dynamic programming and Monte-Carlo methods in
the sense that it works by learning from experience without the need of a
model, but does also use bootstrapping (Sutton and Barto 1998). TD algorithms
are on-line which make them applicable to a wider range of problems as they do
not rely on completing an entire episode prior to computing the updates. After
each _action_ , the past and current experience $s$ and $s^{\prime}$ are used
to compute the error of _value_ estimates $\hat{v}_{\theta}(s)$. The following
Equation defines the _TD-error_ $\delta$ for _state-value_ prediction that is
used in TD-learning. It is a measure of how accurately the reward received at
a particular time-step was predicted (i.e. RPE).

$\delta=r+\gamma\hat{v}_{\theta}(s^{\prime})-\hat{v}_{\theta}(s)$

#### Actor-Critic Algorithm.

PG methods are used in environments with _continuous-state_ spaces. The
_policy_ (actor) is in this case a parameterized function $\pi_{\theta}$.
Parameters are in most cases the weights of a neural network. By constructing
a performance measure $J(\pi_{\theta})$ and taking its gradient with respect
to the parameters $\theta$, the policy $\pi_{\theta}$ is trained using
gradient ascent. This approach learns a policy $\pi_{\theta}$, without the
need of a _value_ -function (critic) $\hat{v}_{\theta}$. Nonetheless, most
successful policy-gradient algorithms, such as actor-critic, learn a _value_
-function $\hat{v}_{\theta}$, too, for faster convergence.

In _actor-critic (AC)_ algorithms the actor computes the _actions_ , and the
critic evaluates the actor’s decisions. In particular, the value
$\hat{v}_{\theta}(s)$ of the current state $s$, is compared to its value
$r\,{+}\,\gamma\hat{v}_{\theta}(s^{\prime})$ after taking an action $a$, that
is, after improving its knowledge by getting a reward $r$ and moving to state
$s^{\prime}$. This leads to the the TD-error discussed above.

The TD-error is a measure for the accuracy of the _reward-prediction_ , acting
as RPE. Given its importance, it is used to update both the actor and the
critic, by acting as a reinforcement signal (Sutton and Barto 1998). Note the
difference between the _reward-signal_ $r$ and the _reinforcement-signal_
$\delta$: if the reward $r$ is predicted perfectly by $\hat{v}_{\theta}$, no
reinforcement $\delta$ takes place, whereas the absence of a predicted reward
$r$ leads to a negative reinforcement $\delta$. Although actor-critic was
devised through pure mathematical considerations, the algorithm interestingly,
very much resembles the dopaminergic feedback pathways found in the mamalian
brain.

#### Eligibility Traces (ET).

They were introduced in the context of RL, as a method for approximating
gradients _online_ , that is, during forward computation. In RL tasks, rewards
are often delayed, meaning that they are not given immediately after the
action that was performed to receive them. This is not a challenge for offline
algorithms, such as Monte-Carlo methods, because each weight update is
computed after a complete episode, when the final reward is known. Conversely,
online algorithms need some way of attributing delayed rewards to past
actions. One possibility is to delay the calculations of updates for a fixed
number of steps, which is called $n$-step bootstrapping. However, this
backwards approach comes with significant computational drawbacks.

Another way of factoring-in future rewards is by using eligibility traces
$\mathbf{e}$. The main idea is to keep a running average of each synapse’s
contribution to the network’s output which can be thought of as a _short-term_
memory paralleling the _long-term_ memory represented by the synapse weights.
Eligibility traces can be used to unify and generalize Monte-Carlo and TD
methods. (Sutton and Barto 1998)

Particularly, TD($\lambda$) makes use of eligibility traces $\mathbf{e}$ in
that weight updates are computed by multiplying the trace with the TD-error
$\delta$. By accumulating the gradients of the state-value function, each
weight’s contribution is recorded. The trace $\mathbf{e}$ decays with factor
$\gamma\lambda$ where $\gamma$ is the discount factor:

$\displaystyle\mathbf{e}$ $\displaystyle=\gamma\lambda\
\mathbf{e}+\nabla\hat{v}_{\theta}(s)$ $\displaystyle\theta$
$\displaystyle\leftarrow\theta+\eta\ \delta\ \mathbf{e}$

When using parameterized linear-function approximation
$\hat{v}_{\theta}(x_{t})\,{=}\,w^{\top}x_{t}\,{+}\,b$, with parameters $[w,b]$
as in the original TD($\lambda$), the gradient of the loss with respect to
$[w,b]$ is simply
$\nabla_{[w,b]}L(t)\,{=}\,[x_{t}^{\top}\nabla\hat{v}_{\theta}(s),\
\nabla\hat{v}_{\theta}(s)]$. RNNs in contrast retain a hidden state, also
influenced by the same parameters, making the computation of gradients
cumbersome.

#### Backpropagation Through Time (BPTT).

Computation of gradients in RNNs is more complicated than in MLP feed-forward
architecturess, as the error vector for each layer recurrently depends on all
computations done in the past. Intuitively, by unrolling the RNN, an MLP is
constructed, with as many layers as there are time-steps in the input
sequence. The error signal is then propagated back to the beginning of the
input sequence, hence the name (Werbos 1990).

In practice, (truncated) BPTT has a fixed time-horizon $H$ that determines the
length of dependencies the model will be able to learn, but does also scales
computational complexity.

#### Real-Time Recurrent Learning (RTRL).

This method trains RNNs by computing the updates _online_. This is made
possible by introducing explicit feedback connections, and computing an
estimate of the _error-vector_ during the feedforward computation. By using an
estimate only, the update at each step deviates from the true gradients, which
can be mitigated by a small enough learning rate $\eta$. One big advantage of
RTRL-based algorithms, is that the computation time of an update-step is
constant in the number of task steps. Nonetheless, RTRL has a worse
computation complexity than BPTT and is therefore not used in practice. The
update rules used in RTRL are derived as follows.

Given a dataset consisting of the multivariate time-series
$x_{t}\,{\in}\,\mathbb{R}^{I}$ of inputs and $y_{t}\,{\in}\,\mathbb{R}^{O}$ of
labels, we want to minimize some loss function
$\mathcal{L}_{\theta}=\sum_{t=0}^{T}L_{\theta}(y_{t})$ by gradient descent.
This is achieved by taking small steps in the direction of the negative
gradient of the total loss:

$\Delta\theta=-\eta\nabla_{\theta}\mathcal{L}_{\theta}=-\eta\sum\limits_{t=0}^{T}\Delta\theta(t)=-\eta\sum_{t=0}^{T}\nabla_{\theta}L_{\theta}(y_{t})$

We can compute the gradient of the loss at each individual step as
$\Delta\theta(t)=\nabla_{\theta}L_{\theta}(t)=\nabla_{\theta}\hat{y}_{t}\nabla_{\hat{y}_{t}}L_{\theta}(t)$
with $\hat{y}_{t}$ being the output of the RNN at timestep $t$. A common
choice in supervised learning is the MSE
$L_{\theta}(t)=\frac{1}{2}\langle\varepsilon_{t},\varepsilon_{t}\rangle$ where
$\varepsilon_{t}=\hat{y}(t)-y(t)\in\mathbb{R}^{O}$ and
$\nabla_{\hat{y}_{t}}L_{\theta}(t)=\varepsilon_{t}$. When employing an RNN
with linear output mapping the gradient of the model output can be further
expanded into
$\nabla_{\theta_{R}}\hat{y}_{t}=\nabla_{\theta_{R}}h_{t}\nabla_{h_{z}}\hat{y}_{t}$.
RTRL is defined for the special class of ODE RNNs below.

#### ODE RNNs.

A recent type of RNNs that enjoy great popularity are ODE-based RNNs. They
take the form $h_{t+1}=h_{t}\,{+}\,f(x_{t},h_{t})$, such as CT-RNNs (Funahashi
and Nakamura 1993) and NeuralODEs (Chen et al. 2019). These models integrate
an ODE $f(x_{t},h_{t})\,{=}\,\frac{\partial x_{t}}{\partial dt}$. Hence, ODE-
RNNs can be evaluated at continuous time steps. This can be helpful, when
dealing with missing or irregularly-sampled data.

We can calculate the gradient of the ODE-RNN unit’s activation with respect to
the parameters $\theta$, recursively. To this end, we define the _immediate
Jacobian_ $\bar{J}_{t}\,{=}\,\nabla_{\theta}f(x_{t},h_{t})$, with
$\nabla_{\theta}$ being the partial derivative with respect to $\theta$, and
the approximate Jacobian _trace_ $\hat{J}_{t}$ where we will also make use of
the total derivative $\frac{d}{d\theta}$.

Figure 2: Schematics showing how gradients are passed back to each state of
the RNN. The network consists of Linear TD($\lambda$) and a shared RNN body.
Gradients with respect to actor and critic losses are propagated back towards
$h_{t}$ and $h_{t-1}$ respectively.

$\displaystyle\hat{J}_{t+1}$
$\displaystyle:=\frac{d}{d\theta}h_{t+1}=\frac{d}{d\theta}\left(h_{t}+f(x_{t},h_{t})\right)$
(1) $\displaystyle=\frac{d}{d\theta}h_{t}+\frac{d}{d\theta}f(x_{t},h_{t})$
$\displaystyle=\hat{J}_{t}+\nabla_{\theta}f(x_{t},h_{t})+\frac{d}{d\theta}h_{t}\nabla_{h_{t}}f(x_{t},h_{t})$
$\displaystyle=\hat{J}_{t}+\bar{J}_{t}+\hat{J}_{t}\nabla_{h_{t}}f(x_{t},h_{t})$
$\displaystyle=\hat{J}_{t}\left(\mathbb{I}+\nabla_{h_{t}}f(x_{t},h_{t})\right)+\bar{J}_{t}$

The last row defines the Jacobian trace recurrently, in terms of the immediate
Jacobian and a linear combination of the past trace
$\hat{J}_{t}\nabla_{h_{t}}f(x_{t},h_{t})$, allowing to calculate it in
parallel to the forward computation of the RNN. When taking an optimization
step we can calculate the actual gradients as
$\Delta\theta(t)=\hat{J}_{t}\nabla_{h_{t}}L_{\theta(t)}=\hat{J}_{t}W_{out}^{\top}\varepsilon_{t}$.

Note that RTRL does not require separate phases for forward and backward
computation and it is therefore biologically plausible in the time domain.
However, RTRL still assumes perfect communication of gradient information
among all neurons in the network. Particularly, the error signal
$W^{\top}\varepsilon_{t}$ assumes _weight transport_ and so do the gradients
communicated within the RNN $\hat{J}_{t}\nabla_{h_{t}}f(x_{t},h_{t})$.

In general, RTRL has complexity $\mathcal{O}(n^{4})$ in the number of neurons
$n$ compared to $\mathcal{O}(n^{2})$ for BPTT (considering fixed horizon).
However, a computationally more efficient version for sparse RNNs was proposed
in (Menick et al. 2020).

#### Random-Feedback Local Online Learning (RFLO).

A biologically plausible descendant of RTRL is RFLO (Murray 2019). This
leverages the ODE of simple CT-RNNs in order to simplify the RTRL update
substantially, dropping all parts that are biologically implausible. RFLO
improves biological plausibility of RTRL in two ways:

  * •

Weight transport during error backpropagation is avoided by using feedback
alignment.

  * •

Locality of gradient information is ensured by dropping all non-local terms
from the gradient computation.

To go over the equations describing RFLO we must first define continuous-time
recurrent neural networks.

#### Continuous-Time RNN (CT-RNN).

For modelling time-series data $\\{x_{t}\,{:}\,t\,{\in}\,\mathbb{N}_{0}\\}$, a
neural network is required, capable of retaining information about previous
inputs. This was historically addressed by RNNs like the Elman RNN, consisting
of a recurrent linear layer with linear input mapping and $\tanh$ activation
$\varphi$, followed by linear output mapping:

$\displaystyle h_{t+1}$ $\displaystyle=\varphi(W_{I}x_{t}+W_{R}h_{t}+b)$
$\displaystyle y_{t}$ $\displaystyle=W_{out}h_{t}+b_{out}$

CT-RNNs (Funahashi and Nakamura 1993) compute $h_{t}$ by adding the nonlinear
term scaled by time constant $\frac{1}{\tau}$ to $h_{t-1}$, and subtracting
the regularization term $-\frac{1}{\tau}h_{t-1}$:

$\displaystyle h_{t+1}$
$\displaystyle=h_{t}+\tau^{-1}\left(-h_{t}+\varphi(W_{I}x_{t}+W_{R}h_{t}+b)\right)$

CT-RNNs are ODE-RNNs as defined above and can be interpreted as ODEs with
dynamics:

$\dot{h}=\tau^{-1}\left(-h_{t}+\varphi(W_{I}x_{t}+W_{R}h_{t}+b)\right)$

The time-difference version is simply the one-step Euler expansion. The
expression for the state update tightly resembles conductance-based models of
the membrane potentials in biological neurons. (Gerstner et al. 2014)

#### A Conveniently-Simplified CT-RNN.

The simplest CT-RNN conceivable has $N$ hidden states and $I$ inputs,
activation $\varphi$ and a combined weight matrix $W\in\mathbb{R}^{N\times X}$
where $Z\,{=}\,I\,{+}\,N\,{+}\,1$. Each neuron has a time-constant
$\tau\,{\in}\,\mathbb{R}^{N}$ and the next state $h_{t+1}\in\mathbb{R}^{N}$ is
computed as follows:

$h_{t+1}=h_{t}+\frac{1}{\tau}\left(-h_{t}+\varphi(W\xi_{t})\right)\qquad\xi_{t}=\begin{bmatrix}x_{t}\\\
h_{t}\\\ 1\end{bmatrix}\in\mathbb{R}^{Z}$

where $x_{t}$ is the input at time $t$ and $1$ concatenated to $\xi_{t}$
accounts for the bias. The output $\hat{y}_{t}\in\mathbb{R}^{O}$ is given by a
linear mapping $\hat{y}_{t}=W_{out}h_{t}$. When interpreting the difference
equation as an ODE then $\tau\dot{h}=-h_{t}+\varphi(W\xi_{t})$.

RFLO leverages the state-update expression in order to simplify the RTRL
update. For this we expand the gradient of $f$ in equation 1. Note that
previous work on RFLO kept the time constant $\tau$ fixed and trained the
recurrent weights $W$ only, hence the restricted $\hat{J}^{W}$ in the
equation:

$\displaystyle\hat{J}_{t+1}^{W}=$ $\displaystyle\
\frac{d}{dW}h_{t}+\frac{d}{dW}f(x_{t},h_{t})$ (2) $\displaystyle=$
$\displaystyle\
\frac{d}{dW}h_{t}+\frac{d}{dW}\frac{1}{\tau}\left(-h_{t}+\varphi(W\xi_{t})\right)$
$\displaystyle=$ $\displaystyle\
\frac{d}{dW}h_{t}(1-\frac{1}{\tau})+\frac{d}{dW}\frac{1}{\tau}\varphi(W\xi_{t})$
$\displaystyle=$ $\displaystyle\ (1-\frac{1}{\tau})\
\hat{J}_{t}^{W}+\frac{1}{\tau}\nabla_{W}\varphi(W\xi_{t})$
$\displaystyle\quad+\frac{1}{\tau}\hat{J}_{t}^{W}\nabla_{h_{t}}\varphi(W\xi_{t})$

In order to achieve biological plausibility, RFLO boldly drops the last
summand, since it requires horizontal gradient communication. The partial
derivative of the activation is simply
$\nabla_{W}\varphi(W\xi_{t})=\varphi^{\prime}(W\xi_{t})^{\top}\xi_{t}$ where
$\varphi^{\prime}$ is the point-wise derivative of the activation function
$\varphi$:

$\hat{J}_{t+1}^{W}\approx(1-\frac{1}{\tau})\
\hat{J}_{t}^{W}+\frac{1}{\tau}\varphi^{\prime}(W\xi_{t})^{\top}\xi_{t}$ (3)

Finally, to avoid weight transport, parameter updates are computed using an
upstream error vector that is propagated back using feedback alignment. The
parameters updates are computed as $\Delta
W(t)=\hat{J}_{t}^{W}B\varepsilon_{t}$ making use of a fixed random matrix $B$.
Surprisingly, learning is still achievable with this simplified version of
RTRL, as has been shown by Murray (2019); Marschall, Cho, and Savin (2020).

While RTRL has complexity $\mathcal{O}(n^{4})$ in the number of neurons $n$,
RFLO has the same complexity as backpropagation $\mathcal{O}(n^{2})$. The
reader is referred to (Murray 2019) and Marschall, Cho, and Savin (2020) for
detailed theoretical comparisons between RTRL and RFLO.

Algorithm 1 Real-Time Recurrent Reinforcement Learning

0: Linear policy: $\pi(a|h,\theta_{A})$

0: Linear value function: $\hat{v}(h,\theta_{C})$

0: CT-RNN body: RNN$([o,a,r],h,\hat{J},\theta_{R})$

$\theta_{A},\theta_{C},\theta_{R}\leftarrow$ Randomly initialize parameters

if RFLO then

$B_{A},B_{C}\leftarrow$ Randomly initialize feedback matrices

end if

$h,z_{A},z_{C},z_{h}\leftarrow\mathbf{0}$

$I\leftarrow 1$

$o\leftarrow$ Reset Environment

$h,\hat{J}\leftarrow$RNN$([o,\mathbf{0},0],h,\theta_{R})$ $\triangleright$
RFLO to compute $\hat{J}$

$v\leftarrow\hat{v}(h,\theta_{C})$

while not done do

logits $\leftarrow\pi(h,\theta_{A})$

$a\leftarrow$ Sample(logits)

$o,R\leftarrow$ Take action $a$ in Environment

$h^{\prime},\hat{J}^{\prime}\leftarrow$RNN($[o,a,r],h,\hat{J},\theta_{R}$)

$v^{\prime}\leftarrow\hat{v}(h^{\prime},\theta_{C})$

$\delta\leftarrow R+\gamma v^{\prime}-v$ $\triangleright$ TD error

$g_{C}\leftarrow B_{C}\mathbf{1}$ $\triangleright$ feedback alignment

$g_{A}\leftarrow B_{A}\nabla_{\text{logits}}[\ln\pi[a]+\eta_{H}$H$(\pi)]$

$z_{h}\leftarrow\gamma\lambda_{R}z_{R}+g_{C}+g_{A}$ $\triangleright$
eligibility traces

$z_{C}\leftarrow\gamma\lambda_{C}z_{C}+\nabla_{\theta_{C}}\hat{v}$

$z_{A}\leftarrow\gamma\lambda_{A}z_{A}+\nabla_{\theta_{A}}[\ln\pi[a]+\eta_{H}$H$(\pi)]$

$\theta_{C}\leftarrow\theta_{C}+\alpha_{C}\delta z_{C}$ $\triangleright$
parameter updates

$\theta_{A}\leftarrow\theta_{A}+\alpha_{A}\delta z_{A}$

$\theta_{R}\leftarrow\theta_{R}+\alpha_{R}\hat{J}\delta z_{h}$

$v\leftarrow v^{\prime},\quad h\leftarrow
h^{\prime},\quad\hat{J}\leftarrow\hat{J}^{\prime},\quad I\leftarrow\gamma I$

end while

## Real-Time Recurrent Reinforcement Learning

We propose a fully online reinforcement learning algorithm that builds on RFLO
and TD($\lambda$) we call RTRRL. Algorithm 1 shows the outline of the method.
Intuitively, the approximate Jacobian $\hat{J}_{t}$ is computed at each RNN
step, and combined with the TD-error of TD($\lambda$) to update the RNN
weights. We follow the Meta-RL approach delineated in (Wang et al. 2018)
feeding in the previous action and reward to the RNN body alongside the
observation at each step. $\pi$ and $\hat{v}$ are the Actor and Critic
functions parameterized by $\theta_{A}$ and $\theta_{C}$. The gradients of
their outputs are passed back to the single layer RNN, parameterized by
$\theta_{R}$, through random feedback matrices $B_{A}$ and $B_{C}$
respectively. We train the Actor and Critic using TD($\lambda$) and take small
steps in direction of the action probability and value estimate respectively.
The gradients for each function are accumulated by means of additive
eligibility traces $z_{A,C}$ with $\lambda$ decay. For the actor, we also
compute the gradient of the action distribution’s entropy $H(\phi)$. A third,
combined eligibility trace $z_{R}$ for the gradients of action probability and
value prediction error is maintained for the RNN. When using RFLO the passing
back of the gradients is done via randomly initialized fixed backwards
matrices.

The jacobian can also be computed using RTRL which reduces variance in the
gradients at a higher computational cost. We can also choose to propagate back
to the recurrent layer by using the forward weights as in backpropagation.
However, we found that the biologically plausible feedback alignment works
just as well.

#### Neuron Model.

We use the simplified CT-RNN of the previous section for the RNN body, where
we compute the Jacobian $\hat{J}_{t}$ by using RTRL or RFLO as explained
above. On top of that, we also derive the approximate Jacobian for $\tau$.

$\displaystyle\hat{J}_{t+1}^{\tau}=$ $\displaystyle\
\hat{J}_{t}^{\tau}-\frac{d}{d\tau}\frac{1}{\tau}h_{t}+\frac{1}{\tau}\varphi(W\xi_{t})$
(4) $\displaystyle=$ $\displaystyle\
\hat{J}_{t}^{\tau}-\nabla_{\tau}\frac{1}{\tau}h_{t}-\hat{J}_{t}^{\tau}\nabla_{h_{t}}\frac{1}{\tau}h_{t}+\frac{d}{d\tau}\frac{1}{\tau}\varphi(W\xi_{t})$
$\displaystyle=$ $\displaystyle\
\hat{J}_{t}^{\tau}(1-\frac{1}{\tau})+\frac{1}{\tau^{2}}h_{t}+\nabla_{\tau}\frac{1}{\tau}\varphi(W\xi_{t})$
$\displaystyle\quad+\hat{J}_{t}^{\tau}\nabla_{h_{t}}\frac{1}{\tau}\varphi(W\xi_{t})$
$\displaystyle=$ $\displaystyle\
\hat{J}_{t}^{\tau}(1-\frac{1}{\tau})+\frac{1}{\tau^{2}}h_{t}-\frac{1}{\tau^{2}}\varphi(W\xi_{t})$
$\displaystyle\quad+\hat{J}_{t}^{\tau}\nabla_{h_{t}}\frac{1}{\tau}\varphi(W\xi_{t})$

Again we drop the communicated gradients
$\hat{J}_{t}^{\tau}\nabla_{h_{t}}\varphi(W\xi_{t})$ and arrive at the
expression for $\hat{J}^{\tau}$.

$\displaystyle\hat{J}_{t+1}^{\tau}\approx\hat{J}_{t}^{\tau}(1-\frac{1}{\tau})+\frac{1}{\tau^{2}}\left(h_{t}-\varphi(W\xi_{t})\right)$
(5)

#### Network structure.

The fully connected RNN layer acts as a filter in classical control theory,
and estimates the latent state of the POMDP. Linear Actor and critic functions
then take the latent state as input and compute action and value respectively.
A more detailed explanation of our implementation including a list of
hyperparameters is given in the Appendix.

Model | Linear TD($\lambda$) | RTRL CT-RNN | RFLO CT-RNN | PPO CT-RNN | PPO LSTM
---|---|---|---|---|---
Environment |  |  |  |  |
BernoulliBandit | 68.88$\pm$0.90 | 89.78$\pm$0.22 | 89.34$\pm$0.66 | 89.14$\pm$18.54 | 89.38$\pm$0.24
MetaMaze | 2.80$\pm$2.34∗ | 12.00$\pm$2.68 | 10.00$\pm$3.13 | 10.80$\pm$1.34 | 10.40$\pm$1.04
DiscountingChain | 1.10$\pm$0.04 | 1.04$\pm$0.04 | 1.10$\pm$0.04 | 1.10$\pm$0.00 | 1.10$\pm$0.00
UmbrellaChain | 1.08$\pm$0.47 | 1.19$\pm$0.53 | 0.37$\pm$0.50 | 1.04$\pm$0.12 | 1.03$\pm$0.12
MntnCarCont | -26.63$\pm$17.59 | -79.98$\pm$11.72 | -88.29$\pm$11.80 | -1119.82$\pm$319.57 | -1114.49$\pm$322.89
Acrobot | -500.00$\pm$0.00 | -80.97$\pm$187.54 | -86.72$\pm$227.18 | -471.40$\pm$221.84 | -84.14$\pm$180.65
CartPole | 384.62$\pm$75.31 | 500.00$\pm$205.13 | 500.00$\pm$0.00 | 500.00$\pm$0.00 | 500.00$\pm$0.00
MntnCarCont-vel | -41.87$\pm$12.61 | -2.60$\pm$25.95 | -35.88$\pm$30.43 | -1079.70$\pm$204.52 | -1126.11$\pm$244.08
MntnCarCont-pos | -43.46$\pm$11.84 | -1.16$\pm$31.60 | -59.09$\pm$37.75 | -1071.67$\pm$40.27 | -1077.49$\pm$128.45
CartPole-vel | 454.55$\pm$26.52 | 500.00$\pm$0.00 | 500.00$\pm$0.00 | 112.11$\pm$40.38 | 138.37$\pm$31.41
CartPole-pos | 55.87$\pm$1.18 | 137.36$\pm$177.76 | 500.00$\pm$122.30 | 70.76$\pm$11.42 | 56.15$\pm$82.74
DeepSea-16 | -0.00$\pm$0.42 | 0.99$\pm$0.00 | 0.99$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
DeepSea-8 | 0.99$\pm$0.48 | 0.99$\pm$0.00 | 0.99$\pm$0.42 | 0.99$\pm$0.00 | 0.00$\pm$0.54
DeepSea-4 | 0.99$\pm$0.00 | 0.99$\pm$0.42 | 0.99$\pm$0.31 | 0.99$\pm$0.00 | 0.99$\pm$0.00
MemoryChain-64 | 0.24$\pm$0.06 | 0.22$\pm$0.06 | 0.22$\pm$0.06 | 1.00$\pm$0.44 | 0.09$\pm$0.10
MemoryChain-32 | 0.17$\pm$0.04 | 1.00$\pm$0.41 | 0.16$\pm$0.07 | 1.00$\pm$0.40 | 0.07$\pm$0.41
MemoryChain-16 | 0.12$\pm$0.04 | 1.00$\pm$0.36 | 0.87$\pm$0.42 | 1.00$\pm$0.00 | 1.00$\pm$0.00
MemoryChain-8 | 0.07$\pm$0.03 | 1.00$\pm$0.00 | 1.00$\pm$0.28 | 1.00$\pm$0.00 | 1.00$\pm$0.00
MemoryChain-4 | 0.07$\pm$0.01 | 1.00$\pm$0.00 | 1.00$\pm$0.00 | 1.00$\pm$0.00 | 1.00$\pm$0.00
Table 1: Summary of RL experiments: Numbers reported are the median reward and
standard deviation for 5 runs. In each case, the best validation reward
achieved throughout training was used. ∗Due to problems with divergence, we
had to run a linear TD($\lambda$) version on MetaMaze, with a severely reduced
learning rate of $10^{-5}$.

## Experiments

We evaluate the feasibility of our recurrent online actor-critic algorithm by
testing it on common RL benchmarks provided by the gymnax package (Lange
2022). The tasks comprise fully and partially observable MDPs, each with
discrete and continuous actions. As baselines we consider TD($\lambda$) with
linear function approximation and PPO with BPTT for LSTM and CT-RNN. Our
implementation of PPO is based on purejaxrl (Lu et al. 2022). For each
environment, we trained for a maximum of 50 million steps or until 20
subsequent epochs showed no improvement. Our results show that RFLO often can
outperform both RTRL and BPTT. Table 1 summarizes all our results for the
experiments explained in the remainder of this section. The column ”RFLO CT-
RNN” corresponds to RTRRL as defined above, but we also included results for
RTRL. The values shown are the median and standard-deviation of the best
evaluation episodes experienced throughout the training of 5 runs per
environment.

### Types of Action Spaces

#### Discrete Actions.

We report results for experiments on the classical CartPole and Acrobot
environments, but also for the bsuite environments (Osband et al. 2020), that
are specifically tailored to test individual aspects of RL tasks. For discrete
actions, the outputs of the actor are the log probabilities for each action,
and past actions fed to the RNN are represented in one-hot encoding.

#### Continuous Actions.

To obtain a stochastic policy, a common trick is to use a probability
distribution parameterized by the model output. A natural choice is to use a
Gaussian whose mean and standard-deviation are outputs of the neural network.
Assuming i.i.d. action components, the network output has to be in
$\mathbb{R}^{2U}$ where $U$ is the size of the action vector. An action is
obtained by sampling from the distribution. Reparameterization allows for
obtaining the probability of the selected continuous action and subsequently
computing the gradient of $\log\pi[a]$ in policy-gradient algorithms.

Using reparameterization we are able to apply our RTRRL algorithm to
continuous-control tasks, too. In the table we provide results for
MountainCarContinuous, a continuous version of the classical MountainCar
example (Sutton and Barto 1998).

### Partial Observability

#### Masked Observations.

We took the classical CartPole MDP and transformed it into a POMDP by masking
out parts of the observation. We consider observations that only contain
positions $(x,\theta)$ or velocities $(\dot{x},\dot{\theta})$ called CartPole-
pos and CartPole-vel respectively. The same was done with
MountainCarContinuous. Our results show that, when observing positions only,
RFLO is the only method able to solve the task.

#### Memory Length.

We test the memory capacity of our RTRRL models by learning to remember the
state of a bit for an extended number of steps. The MemoryChain environment
can be thought of as a T-maze experiment, in which when conducted with mice, a
long corridor leads up to a T-section. Each mouse is given a queue at the
entrance to the corridor, indicating on which path leaving the T-section it
will find food (Osband et al. 2020).

The experiment tests if the model can remember the state of a bit for a fixed
number of time steps. Increasing the number of steps increases difficulty. We
conducted MemoryChain experiments with 32 neurons for exponentially increasing
memory lengths. The results show that using approximate gradients hampers
memory capacity. Surprisingly, CT-RNNs trained with RTRL outperform LSTMs of
the same size that were trained with BPTT.

#### Deep Exploration.

Exploration versus exploitation is a trade-off, central to all agents learning
a task online. The DeepSea environment included in bsuite is tailored to
benchmark the ability of RL algorithms to explore in unfavourable
environments. At each step, the agent has to decide on one of two actions
which randomly map to left and right in a way that is fixed at each position.
A small negative reward is given when the action leading right was chosen and
a large positive reward is given at the end of each episode, if the agent
chose right each time. The agent is required to explore to reach the rightmost
position albeit receiving a negative reward when moving towards it. Again, we
give results for exponentially increasing task length.

The classic control environment Acrobot, too, requires extensive exploration.
A double pendulum starting in hanging position is set into motion by
controlling the middle joint. The outer segment has to be elevated above a
certain height, located above the anchor of the inner segment. Figure 3 shows
the median rewards of 5 runs over the number of environment steps. We find
that RTRRL with RFLO finds a solution significantly faster than the other
methods hinting at superior exploration of our algorithm. We postulate that
this is due to the initially erroneous gradients introduced by feedback
alignment and the simplified RTRL update. It has to be be noted that RFLO with
fixed $\tau$ performed better here, which further deepens our belief that less
accurate gradients lead to improved exploration.

Figure 3: Median reward of 5 runs on Acrobot. The key to solving this
environment is exploration. RNNs trained with RFLO solve it quicker than when
trained with BPTT.

## Conclusion

We introduced a novel, biological plausible, recurrent RL algorithm called
real-time recurrent reinforcement learning (RTRRL), by combining TD($\lambda$)
with online gradient computation. We empirically demonstrate its viability on
a wide range of fully and partially observable MDPs. Two types of gradient
computation with differing degree of biological plausibility, namely RTRL and
RFLO, are compared to BPTT and each other. Our results show, that using
approximate gradients for updating model parameters still finds satisfactory
solutions, and in some cases, it even improves on state of the art.
Particularly, we found that RTRRL outperforms PPO with BPTT, when solving
tasks requiring exploration in unfavourable environments.

Although the results presented in this paper are empirically convincing, some
limitations have to be discussed. When not using batched experiences, our
algorithm unsurprisingly suffers from large variance and in some cases
catastrophic forgetting ensues. We are able to mitigate this problem by using
an exponentially decaying learning rate but for simplicity we chose to make
our experiments without this improvement. Since a biologically plausible agent
should retain its capability of reacting to shifts in the environment, the
adaptability of our algorithm would be tainted as the learning rate of the RNN
approaches 0. Nonetheless, RTRRL most of the time converges to an optimal
solution even without a decaying learning rate. Preliminary results on more
complex environments with continuous actions such as Mujoco (Todorov, Erez,
and Tassa 2012) showed less convincing results, suggesting that the algorithm
is best suited for discrete actions. An RTRRL version of deep deterministic
policy gradient (DDPG) is one possible venue of further investigation to
better address continuous actions.

RTRRL is grounded in neuroscience and adequately explains how biological
neural networks learn to act in unknown environments. The network structure
resembles the interplay of dorsal and ventral striatum located in the basal
ganglia, featuring global RPEs found in dopaminergic pathways projecting from
the ventral tegmental area and the substantia nigra zona compacta to the
striatum and cortex. (Wang et al. 2018) The role of dopamine as RPE was
established experimentally, where it was shown that upon receiving an
unexpected reward, dopamine is released subsequently reinforcing the behavior
that led to the reward.

However, if the reward is preceded by a conditioned stimulus, it is released
as soon as the stimulus occurs instead of when receiving the expected reward.
If then the expected reward is absent, dopamine levels will drop below
baseline representing a negative reinforcement signal (Wise 2004).
Dopaminergic synapses are usually located at the dendritic stems of glutamate
synapses (Kandel 2013) and can therefore effectively mediate synaptic
plasticity.

More specifically, the _ventral striatum_ would correspond to the _critic_ in
RTRRL and the _dorsal striatum_ to the _actor_ , with dopamine axons targeting
both the ventral and dorsal compartmens (Sutton and Barto 1998). The axonal
tree of dopaminergic synapses is represented by the backward weights in RTRRL
($B$ when using RFLO, $W_{out}^{\top}$ when using RTRL). Dopamine subsequently
encodes the _TD-error_ as RPE, which is used to update both the _actor_ and
the _critic_.

In a noteworthy departure from contemporary DRL techniques, RTRRL stands out
as being fully online, and eliminating the necessity for weight transport and
horizontal gradient communication.

## References

  * Bartunov et al. (2018) Bartunov, S.; Santoro, A.; Richards, B. A.; Marris, L.; Hinton, G. E.; and Lillicrap, T. 2018.  Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures.  _arXiv:1807.04587 [cs, stat]_.
  * Behrens et al. (2018) Behrens, T. E. J.; Muller, T. H.; Whittington, J. C. R.; Mark, S.; Baram, A. B.; Stachenfeld, K. L.; and Kurth-Nelson, Z. 2018.  What Is a Cognitive Map? Organizing Knowledge for Flexible Behavior.  _Neuron_ , 100(2): 490–509.
  * Chen et al. (2019) Chen, R. T. Q.; Rubanova, Y.; Bettencourt, J.; and Duvenaud, D. 2019.  Neural Ordinary Differential Equations.  _arXiv:1806.07366 [cs, stat]_.
  * Chung (2019) Chung, W. 2019.  Two-Timescale Networks for Nonlinear Value Function Approximation.  https://era.library.ualberta.ca/items/d0665ad5-a222-4911-a20e-72d4f7916821.
  * Funahashi and Nakamura (1993) Funahashi, K.-i.; and Nakamura, Y. 1993.  Approximation of Dynamical Systems by Continuous Time Recurrent Neural Networks.  _Neural Networks_ , 6(6): 801–806.
  * Gerstner et al. (2014) Gerstner, W.; Kistler, W. M.; Naud, R.; and Paninski, L. 2014.  _Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition_.  Cambridge: Cambridge University Press.  ISBN 978-1-107-44761-5.
  * Javed et al. (2023) Javed, K.; Shah, H.; Sutton, R.; and White, M. 2023.  Scalable Real-Time Recurrent Learning Using Sparse Connections and Selective Learning.  arxiv:2302.05326.
  * Johard and Ruffaldi (2014) Johard, L.; and Ruffaldi, E. 2014.  A Connectionist Actor-Critic Algorithm for Faster Learning and Biological Plausibility.  In _2014 IEEE International Conference on Robotics and Automation (ICRA)_ , 3903–3909. Hong Kong, China: IEEE.  ISBN 978-1-4799-3685-4.
  * Kandel (2013) Kandel, E. R. 2013.  _Principles of Neural Science_.  New York; Toronto: McGraw-Hill Medical.  ISBN 978-0-07-139011-8.
  * Ladosz et al. (2022) Ladosz, P.; Ben-Iwhiwhu, E.; Dick, J.; Ketz, N.; Kolouri, S.; Krichmar, J. L.; Pilly, P. K.; and Soltoggio, A. 2022.  Deep Reinforcement Learning With Modulated Hebbian Plus Q-Network Architecture.  _IEEE Transactions on Neural Networks and Learning Systems_ , 33(5): 2045–2056.
  * Lange (2022) Lange, R. T. 2022.  gymnax: A JAX-based Reinforcement Learning Environment Library.
  * Lillicrap et al. (2014) Lillicrap, T. P.; Cownden, D.; Tweed, D. B.; and Akerman, C. J. 2014.  Random Feedback Weights Support Learning in Deep Neural Networks.  _arXiv:1411.0247 [cs, q-bio]_.
  * Lu et al. (2022) Lu, C.; Kuba, J.; Letcher, A.; Metz, L.; Schroeder de Witt, C.; and Foerster, J. 2022.  Discovered Policy Optimisation.  _Advances in Neural Information Processing Systems_ , 35: 16455–16468.
  * Marschall, Cho, and Savin (2020) Marschall, O.; Cho, K.; and Savin, C. 2020.  A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks.  _Journal of Machine Learning Research_ , 21(135): 1–34.
  * Menick et al. (2020) Menick, J.; Elsen, E.; Evci, U.; Osindero, S.; Simonyan, K.; and Graves, A. 2020.  Practical Real Time Recurrent Learning with a Sparse Approximation.  In _International Conference on Learning Representations_.
  * Mujika, Meier, and Steger (2018) Mujika, A.; Meier, F.; and Steger, A. 2018.  Approximating Real-Time Recurrent Learning with Random Kronecker Factors.  arxiv:1805.10842.
  * Murray (2019) Murray, J. M. 2019.  Local Online Learning in Recurrent Networks with Random Feedback.  _eLife_ , 8: e43299.
  * Ni, Eysenbach, and Salakhutdinov (2022) Ni, T.; Eysenbach, B.; and Salakhutdinov, R. 2022.  Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs.  arxiv:2110.05038.
  * Ororbia and Mali (2022) Ororbia, A.; and Mali, A. 2022.  Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problems.  arxiv:2209.09174.
  * Osband et al. (2020) Osband, I.; Doron, Y.; Hessel, M.; Aslanides, J.; Sezener, E.; Saraiva, A.; McKinney, K.; Lattimore, T.; Szepesvari, C.; Singh, S.; Roy, B. V.; Sutton, R.; Silver, D.; and Hasselt, H. V. 2020.  Behaviour Suite for Reinforcement Learning.
  * Roth, Kanitscheider, and Fiete (2018) Roth, C.; Kanitscheider, I.; and Fiete, I. 2018.  Kernel RNN Learning (KeRNL).  In _International Conference on Learning Representations_.
  * Sutton and Barto (1998) Sutton, R. S.; and Barto, A. G. 1998.  _Reinforcement Learning: An Introduction_.  Adaptive Computation and Machine Learning. Cambridge, Mass: MIT Press.  ISBN 978-0-262-19398-6.
  * Tallec and Ollivier (2017) Tallec, C.; and Ollivier, Y. 2017.  Unbiased Online Recurrent Optimization.  arxiv:1702.05043.
  * Todorov, Erez, and Tassa (2012) Todorov, E.; Erez, T.; and Tassa, Y. 2012.  MuJoCo: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 5026–5033. IEEE.
  * Wang et al. (2018) Wang, J. X.; Kurth-Nelson, Z.; Kumaran, D.; Tirumala, D.; Soyer, H.; Leibo, J. Z.; Hassabis, D.; and Botvinick, M. 2018.  Prefrontal Cortex as a Meta-Reinforcement Learning System.  _Nature Neuroscience_ , 21(6): 860–868.
  * Werbos (1990) Werbos, P. 1990.  Backpropagation through Time: What It Does and How to Do It.  _Proceedings of the IEEE_ , 78(10): 1550–1560.
  * Williams and Zipser (1989) Williams, R. J.; and Zipser, D. 1989.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks.  _Neural Computation_ , 1(2): 270–280.
  * Wise (2004) Wise, R. A. 2004.  Dopamine, Learning and Motivation.  _Nature Reviews Neuroscience_ , 5(6): 483–494.

Figure 4: RTRRL can be devided into 4 Phases that are repeated throughout
training.

## Appendix

Algorithm 2 repeats RTRRL with if-cases for RTRL without feedback alignment
for demonstrative purposes. Table 2 summarizes the hyperparameters of RTRRL.
We kept them at the listed default values for all our experiments.

### Implementation Details

Our implementation is highly configurable and allows for many tweaks to the
base algorithm. Available options include gradient clipping, learning rate
decay, epsilon greedy policy, delayed RNN parameter updates and much more.
Please refer to table 2 and the Readme in the code folder for a list of
configurables.

The algorithm can be divided into 4 distinct steps that are depicted in figure
4. When developing the algorithm, we had to figure out the proper order of
operations. Figure 5 is a flowchart that was created to help understand at
what point the eligibility traces are combined with the TD-error and
approximate Jacobian, to form the parameter updates.

Description | Symbol | Value
---|---|---
number of neurons | $n$ | 32
discount factor | $\gamma$ | 0.99
Actor learning rate | $\alpha_{A}$ | 1e-3
Critic learning rate | $\alpha_{C}$ | 1e-3
RNN learning rate | $\alpha_{R}$ | 1e-5
entropy rate | $\eta_{H}$ | 1e-5
Actor eligibility decay | $\lambda_{A}$ | 0.8
Critic eligibility decay | $\lambda_{C}$ | 0.8
RNN eligibility decay | $\lambda_{R}$ | 0.8
patience in epochs | 20
maximum environment steps | 50 mil.
optimizer | SGD
batch size | 1
learning rate decay | 0
action epsilon | 0
update period | 1
Table 2: Hyperparameters of RTRRL and values used. Algorithm 2 Real-Time
Recurrent Reinforcement Learning

0: Linear policy: $\pi(a|h,\theta_{A})$

0: Linear value function: $\hat{v}(h,\theta_{C})$

0: CT-RNN body: RNN$([o,a,r],h,\hat{J},\theta_{R})$

$\theta_{A},\theta_{C},\theta_{R}\leftarrow$ Randomly initialize parameters

if RFLO then

$B_{A},B_{C}\leftarrow$ Randomly initialize feedback matrices

end if

$h,z_{A},z_{C},z_{h}\leftarrow\mathbf{0}$

$o\leftarrow$ Reset Environment

$h,\hat{J}\leftarrow$RNN$([o,\mathbf{0},0],h,\theta_{R})$

$v\leftarrow\hat{v}(h,\theta_{C})$

while not done do

logits $\leftarrow\pi(h,\theta_{A})$

$a\leftarrow$ Sample(logits)

$o,R\leftarrow$ Take action $a$ in Environment

$h^{\prime},\hat{J}^{\prime}\leftarrow$RNN($[o,a,r],h,\hat{J},\theta_{R}$)

$v^{\prime}\leftarrow\hat{v}(h^{\prime},\theta_{C})$

$\delta\leftarrow R+\gamma v^{\prime}-v$

$z_{C}\leftarrow\gamma\lambda_{C}z_{C}+\nabla_{\theta_{C}}\hat{v}$

$z_{A}\leftarrow\gamma\lambda_{A}z_{A}+\nabla_{\theta_{A}}[\ln\pi[a]+\eta_{H}$Ent$(\pi)]$

if RTRL then

$z_{h}\leftarrow\gamma\lambda_{R}z_{R}+\nabla_{h}[\hat{v}+\ln\pi[a]+\eta_{H}$Ent$(\pi)]$

else if RFLO then

$g_{C}\leftarrow B_{C}\mathbf{1}$

$g_{A}\leftarrow B_{A}\nabla_{\text{logits}}[\ln\pi[a]+\eta_{H}$Ent$(\pi)]$

$z_{h}\leftarrow\gamma\lambda_{R}z_{R}+g_{C}+g_{A}$

end if

$\theta_{C}\leftarrow\theta_{C}+\alpha_{C}\delta z_{C}$

$\theta_{A}\leftarrow\theta_{A}+\alpha_{A}\delta z_{A}$

$\theta_{R}\leftarrow\theta_{R}+\alpha_{R}\hat{J}\delta z_{h}$

$v\leftarrow v^{\prime},\quad h\leftarrow
h^{\prime},\quad\hat{J}\leftarrow\hat{J}^{\prime}$

end while

Figure 5: Flowchart depicting the computations done throughout one RTRRL step.

### Neuron Model

Figure 6: CT-RNNs are solving an Ordinary Differential Equation. In general,
any solver may be used. When using forward Euler, $dt=k^{-1}$ is a
hyperparameter that determines the number of solver steps and subsequently the
accuracy of the solution.

The simplified CT-RNN outlined in the paper is taken from (Murray 2019). Our
code allows for increasing the number of steps $k=dt^{-1}$ when solving the
underlying Ordinary Differential Equation with the forward Euler method. More
steps leads to a more expressive model, meaning you can get away with fewer
neurons, but also to increased computational complexity. In our experiments we
kept $k=1$ for simplicity.

### Logging

We used Weights&Biases 111https://wandb.ai to log experiment runs. For better
readability, and to keep things open source, we omitted the respective code
from the supplementary material. Upon acceptance, we will publish our code,
including infrastructure for logging large numbers of runs, on GitHub (we are
looking at Aim 222https://aimstack.io/).

## Biological Interpretation

### Dopamine and Learning

Dopamine is a neurotransmitter found in the central nervous system of mammals.
It is widely agreed upon that dopamine plays an important role in rewards and
reinforcement. The molecule is released by specialized neurons located
primarily in two areas of the brain: the substantia nigra zona compacta (SNc)
and the ventral tegmental area (VTA). (Wise 2004) These neurons have large
branching axonal arbors making synapses with many other neurons - mostly in
the striatum and the pre-frontal cortex. Those synapses are usually located at
the dendritic stems of glutamate synapses and can therefore effectively
influence synaptic plasticity. (Sutton and Barto 1998)

Whenever receiving an unexpected reward, dopamine is released which
subsequently reinforces the behavior that led to the reward. However, if the
reward is preceded by a conditioned stimulus, it is released as soon as the
stimulus occurs instead of when receiving the expected reward. If then the
expected reward is absent, levels will drop below baseline representing a
negative reinforcement signal. (Wise 2004) Various experiments have shown that
dopamine is crucial for stamping-in response-reward and stimulus-reward
associations which in turn is needed for motivation when confronted with the
same task in the future. For example, moderate doses of dopamine antagonists
(neuroleptics) will reduce motivation to act. Habitual responses decline
progressively in animals that are treated with neuroleptics. (Wise 2004)

### Ternary Synapses

Hetero-synaptic plasticity is a type of synaptic plasticity that involves at
least three different neurons. Usually, a sensory neurons is forming a synapse
with a motor neuron. A third, so called facilitating neuron also forms a
synapse at the same spot and so is able to influence the signal transmission
between the sensory and the motor neuron. A historic example is found in the
gill-withdrawal reflex circuitry of Aplysia (Kandel 2013). The RPE signalling
facilitating interneuron strengthens the motor response (withdrawing the gill)
when a negative reward (shock) is experienced or expected. RTRRL could be
implemented in biological neural networks with ternary synapses where the
facilitating synapses are projecting back from the TD-error computing
interneuron.

### Brain Structures

Figure 7: The basal ganglia are located at the base of the forebrain and play
a major role in motivation and behavioural learning. Source: Wikimedia,
Leevanjackson444https://commons.wikimedia.org/w/index.php?curid=85845448

There is evidence that certain brain structures could be implementing _actor-
critic_ methods. The striatum is involved in motor and action planning,
decision-making, motivation, reinforcement and reward perception. It is also
heavily innervated by dopamine axons coming from the VTA and the SNc. It is
speculated that dopamine release in the ventral striatum ”energizes the next
response” while it acts by stamping in the procedural memory trace in the
dorsal striatum ”establishing and maintaining procedural habit structures”
(Wise 2004). Subsequently, the ventral striatum would correspond to the
_critic_ and the dorsal striatum to the _actor_ of an RL algorithm (Sutton and
Barto 1998). Dopamine would then correspond to the _TD-error_ which is used to
update both the _actor_ and the _critic_. As dopamine neuron axons target both
the ventral and dorsal striatum, and dopamine appears to be critical for
synaptic plasticity, the similarities are evident. Furthermore, the _TD-error_
and dopamine levels are both encoding the RPE: they are high whenever an
unexpected reward is received and they are low (or negative in case of the
_TD-error_) when an expected reward does not occur. (Sutton and Barto 1998)
These similarities could be beneficial for RL as well as for Neuroscience as
advances in either field could lead to new insights that are beneficial to the
other.

## Plots

Figure 8: Boxplots of the Rewards of 10 runs on MemoryChain per type of
plasticity for increasing memory lengths. BPTT refers to PPO with LSTM.

With the ‘MemoryChain‘ experiments we wanted to answer the question on how
RTRL compares to RFLO and how both approaches compare to actor-critic
algorithms built with BPTT. We reproduce our results in figure 8, this time as
boxplots. We also included a naive approximation to RTRL we call LocalMSE that
only uses the immediate Jacobian for updates. Looking at the boxplot we can
clearly see that RTRL outperforms the other methods and that RFLO is situated
at a middle ground between LocalMSE and RTRL in terms of memory capacity.