# On the cosmological implications of the electroweak vacuum instability:
constraining the non-minimal coupling with inflation

Andreas Mantziris  Department of Physics, Imperial College London, 180 Queen’s
Gate, London, SW7 2AZ, UK<EMAIL_ADDRESS>
###### Abstract

Our current measurements of the Standard Model parameters imply that the Higgs
field resides in a metastable electroweak vacuum, where the vacuum can decay
to a lower ground state, with cataclysmic repercussions for our Universe.
According to our observations, no such event has happened in the observable
universe, in spite of the various energetic processes that could have
triggered it. This work serves as an overview of a method that uses the
metastability of the false vacuum during cosmological inflation to provide
constraints on the Higgs curvature coupling $\xi$. Considering also the
effects of the time-dependent Hubble rate on the effective Higgs potential and
on our past light-cone space-time geometry, results in state-of-the-art lower
$\xi$-bounds from quadratic and quartic chaotic inflation, and Starobinsky-
like power-law inflation.

##  1 Introduction

The experimental values for the parameters of the Standard Model (SM) of
particle physics, especially $m_{h}$ and $m_{t}$ (mass of the Higgs boson and
the top quark), have some important implications for our Universe. First of
all, the Higgs self-interaction $\lambda$ does not diverge below the Planck
scale [1], as it can be seen in the left plot of figure 1. It is evident that
the four-point coupling runs smoothly with the energy scale $\mu$, throughout
the coloured band that signifies the uncertainty in the measurement of the top
quark mass. This suggests that the SM can persist as a valid theory of
fundamental physics at higher energies and thus provide a consistent minimal
model of the early Universe. Secondly, for most of the values within the
$3\sigma$ deviation band, $\lambda$ turns negative during its running. This
leads to the formation of a second, deeper vacuum state at higher values of
the Higgs field $h$, since the Higgs quartic potential is given by

$\displaystyle V_{H}(h,\mu)=\frac{\lambda(\mu)}{4}h^{4}\,,$ (1)

with is a potential barrier separating the two vacua, as shown in the right
plot of figure 1. The presence of a lower ground state renders the EW state,
that our Universe resides in, to be a metastable one because the Higgs field
is prone to vacuum decay. This process can take place via quantum tunnelling,
thermal fluctuations, or a combination of both and because our measurements
dictate that we are still in the EW vacuum, it means that the metastable state
has survived throughout our cosmological history. This realisation inspired us
to study vacuum decay in early universe scenarios to constrain fundamental
physics. More specifically, we derived lower bounds for the Higgs-curvature
coupling $\xi$ from vacuum stability during inflation [2].

Figure 1: Left: running of the Higgs self-coupling, where the coloured bands
correspond to the uncertainties in the top quark mass. Right: double well
potential of a scalar field $\varphi$, with possible vacuum decay trajectories
from a metastable vacuum to the true vacuum state. [3]

##  2 Vacuum decay during inflation

Vacuum decay induces the nucleation of true vacuum bubbles expanding with
velocity close to the speed of light that consume everything in their way.
Since our Universe is still in the EW vacuum, no bubbles could have formed in
our observable universe. This can be expressed in terms of the expectation
value of the number of bubbles in our past light-cone as $\langle{\cal
N}\rangle\leq 1$, where

$\displaystyle d\langle{\cal N}\rangle=\Gamma d{\cal
V}\,\Rightarrow\,\,\langle{\cal N}\rangle=\int_{\begin{subarray}{c}\rm
past\end{subarray}}d^{4}x\sqrt{-g}\Gamma(x)\,,$ (2)

with $\Gamma$ being the decay rate, ${\cal V}$ the space-time volume and $g$
the determinant of the metric.

For our purposes, it is more convenient and intuitive to integrate (2)
backwards in time over the duration of inflation, in terms of $e$-foldings of
inflation $N=\mathrm{ln}\left(a_{\rm inf}/a(\eta)\right)$, i.e.

$\displaystyle\langle{\cal
N}\rangle=\frac{4\pi}{3}\int_{0}^{N_{\mathrm{start}}}dN\left(\frac{a_{\mathrm{inf}}\left(\eta_{0}-\eta\left(N\right)\right)}{e^{N}}\right)^{3}\frac{\Gamma(N)}{H(N)}\,,$
(3)

where $a_{\rm inf}$ is the scale factor at $N_{\rm inf}=0$ (end of inflation),
$\eta_{0}$ the conformal time today and $H$ the Hubble rate. These quantities
are calculated subject to an inflationary potential $V(\phi)$ via

$\displaystyle\frac{d\tilde{\eta}}{dN}$
$\displaystyle=-\tilde{\eta}(N)-\frac{1}{a_{\mathrm{inf}}H(N)}\,,$ (4)
$\displaystyle H^{2}$
$\displaystyle=\frac{V(\phi)}{3M_{P}^{2}}\left[1-\frac{1}{6M_{P}^{2}}\left(\frac{d\phi}{dN}\right)^{2}\right]^{-1}\,,$
(5)

without slow roll approximations, where $\tilde{\eta}=e^{-N}\eta$ and the
inflaton field evolves according to

$\displaystyle\frac{d^{2}\phi}{dN^{2}}$
$\displaystyle=\frac{V(\phi)}{M_{P}^{2}H^{2}}\left(\frac{d\phi}{dN}-M_{P}^{2}\frac{V^{\prime}(\phi)}{V(\phi)}\right)\,.$
(6)

Observations dictate that inflation lasts for at least 60 $e$-folds, i.e.
$N_{\rm start}\geq 60$, and by implementing $\langle{\cal N}\rangle\leq 1$, we
can constrain the remaining free parameter in (3). This is the non-minimal
coupling $\xi$, which enters the calculation via the decay rate $\Gamma$ as
explained below.

The classical solutions to the decay process from false to true vacuum are
famously known as instantons [4, 5]. During inflation, the Hubble rate is at
such high scales that the Hawking-Moss (HM) instanton [6] is the dominant one,
characterised by the decay rate $\Gamma_{\rm HM}$ given by

$\displaystyle\Gamma_{\rm HM}(R)\approx\left(\frac{R}{12}\right)^{2}e^{-B_{\rm
HM}(R)}\,,\quad B_{\rm HM}(R)\approx\frac{384\pi^{2}\Delta V_{\rm
H}}{R^{2}}\,,$ (7)

where $R$ is the Ricci scalar, $B$ the action difference and $\Delta V_{\rm
H}=V_{\rm H}(h_{\rm bar})-V_{\rm H}(h_{\rm fv})$ the height of the potential
barrier, measured from the top of the barrier (bar) to the false vacuum (fv).
We calculate $\Delta V_{\rm H}$ using the Renormalisation Group Improved (RGI)
effective Higgs potential

$\displaystyle V_{\rm H}^{\rm
RGI}(h,R)=\frac{\xi(\mu_{*}(h,R))}{2}Rh^{2}+\frac{\lambda(\mu_{*}(h,R))}{4}h^{4}+\frac{\alpha(\mu_{*}(h,R))}{144}R^{2}\,,$
(8)

where the first is the Higgs-curvature term], the second the self interaction
and the third a radiatively generated gravitational term (in de Sitter (dS)
space) originating from curved space proper renormalisation. The RG scale
$\mu_{*}$ is chosen so that the loop correction to (8) vanishes.

##  3 Results

Solving the system of (3), (4) and (6) for quadratic inflation
$V(\phi)=\frac{1}{2}m^{2}\phi^{2}$, quartic inflation
$V(\phi)=\frac{1}{4}\lambda\phi^{4}$, and Starobinsky-like [7] power-law
inflation
$V(\phi)=\frac{3}{4}\alpha^{2}M_{P}^{4}\left(1-e^{-\sqrt{\frac{2}{3}}\frac{\phi}{M_{P}}}\right)^{2}$
results in the bounds shown in figure 2. As $\xi$ is a running parameter, the
bounds refer to its value at $\mu_{\rm EW}$. In the lower left corner of the
plot, the parameter space is restricted according to the signs of $\lambda$
and $\xi$. If $\lambda$ remains positive during its running, we have stability
of the EW vacuum, and therefore we cannot obtain any constraints from
metastability. The curves terminate at the horizontal dotted line, because our
approach does not take into account any potential destabilization of the Higgs
potential caused by a negative $\xi$ during its RG evolution.

Figure 2: Non-minimal coupling bounds with a varying top quark mass for three
models. [2]

In figure 3, the localised peaks of the integrands of (3) for each
inflationary model highlight the time when bubble production is most likely to
take place. In all cases, the bubbles have nucleated close to the end of
inflation but before the last $e$-folding, while a 0.5 GeV variance around the
central $m_{t}$ value is also accounted for and illustrated with the
corresponding fainter dashed and dotted lines. It is important to underline
the significance of this, because a certain amount of dS approximation has
been used in this computation (see [2] for details) and space-time does not
resemble approximately dS towards the very end of the inflationary expansion.
Moreover the unknown total duration of inflation does not affect these
results, as bubble formation is suppressed unless inflation lasts for an
unrealistic duration of $\sim 10^{60}$ $e$-folds. Thus, these bounds are
trustworthy, within the range of validity of our approximations and
assumptions.

Figure 3: Bubble integrands for different inflationary models and top quark
masses (central value: solid, $\pm 0.5$ GeV: dashed/dotted), with $\xi_{\rm
EW}$ tuned to $\langle{\cal N}\rangle=1$ for each case. [2]

##  4 Conclusions

With the method presented here, we accounted for 1-loop curvature corrections
beyond dS in the RGI effective Higgs potential, to result into model
degenerate, duration independent and $m_{t}$-dependent lower bounds on the
Higgs-curvature coupling, which are summarised as $\xi_{\rm EW}\gtrsim 0.06$.
Furthermore, the dependence of predominant bubble production on the total
duration of inflation hints against eternal inflation, but further study is
required to address this point properly.

This work was conducted under the supervision of Arttu Rajantie and in
collaboration with Tommi Markkanen, with the author being supported by an STFC
PhD studentship.

## References

## References

  * [1] Bednyakov A, Kniehl B, Pikelner A and Veretin O 2015 Phys. Rev. Lett. 115 201802 (Preprint 1507.08833)
  * [2] Mantziris A, Markkanen T and Rajantie A 2021 JCAP 03 077 (Preprint 2011.03763)
  * [3] Markkanen T, Rajantie A and Stopyra S 2018 Front. Astron. Space Sci. 5 40 (Preprint 1809.06923)
  * [4] Callan Curtis G J, Coleman S R and Jackiw R 1970 Annals Phys. 59 42–73
  * [5] Coleman S R and De Luccia F 1980 Phys. Rev. D 21 3305
  * [6] Hawking S and Moss I 1987 Adv. Ser. Astrophys. Cosmol. 3 154–157
  * [7] Starobinsky A A 1979 JETP Lett. 30 682–685


# Neuronal Sequence Models for Bayesian Online Inference

Sascha Frölich<EMAIL_ADDRESS>Psychology Department,
Chair of Neuroimaging,
Technische Universität Dresden  Dimitrije Marković  Psychology Department,
Chair of Neuroimaging,
Technische Universität Dresden  Stefan J. Kiebel  Psychology Department,
Chair of Neuroimaging,
Technische Universität Dresden

###### Abstract

Sequential neuronal activity underlies a wide range of processes in the brain.
Neuroscientific evidence for neuronal sequences has been reported in domains
as diverse as perception, motor control, speech, spatial navigation and
memory. Consequently, different dynamical principles have been proposed as
possible sequence-generating mechanisms. Combining experimental findings with
computational concepts like the Bayesian brain hypothesis and predictive
coding leads to the interesting possibility that predictive and inferential
processes in the brain are grounded on generative processes which maintain a
sequential structure. While probabilistic inference about ongoing sequences is
a useful computational model for both the analysis of neuroscientific data and
a wide range of problems in artificial recognition and motor control, research
on the subject is relatively scarce and distributed over different fields in
the neurosciences. Here we review key findings about neuronal sequences and
relate these to the concept of online inference on sequences as a model of
sensory-motor processing and recognition. We propose that describing
sequential neuronal activity as an expression of probabilistic inference over
sequences may lead to novel perspectives on brain function. Importantly, it is
promising to translate the key idea of probabilistic inference on sequences to
machine learning, in order to address challenges in the real-time recognition
of speech and human motion.

Keywords: neuronal sequences, recurrent neural networks, Bayesian inference,
predictive coding, hierarchies of time scales, review

##  1 Introduction

###  1.1 Sequences in the Brain and Around Us

The world around us and our everyday activities in it are sequentially
structured. For example, the morning routine of making a coffee can be divided
into sequential subroutines of grinding the coffee beans, putting them into
the machine and boiling water. As our own and other people’s movements are
sequentially structured, the continuous stream of information we receive is of
a sequential structure as well. This is for example evident for auditory
input, where speech is based on sequential alterations of frequencies and
amplitudes, which in turn are created by the sequential motion of the vocal
tract. In the visual domain, the same principle applies where someone else’s
movements are inferred by considering their sequential, temporal structure, in
addition to spatial information like posture (Giese and Poggio,, 2003).

Given that both our behaviour and the environment are sequentially structured,
it is not surprising that sequences of neuronal activity have been reported in
a wide range of experimental contexts and seem to play an important role for
various brain functions: To date, the brain structure whose functional
dependence on sequences is most evidently established is the hippocampus, see
(Bhalla,, 2019) for a recent review. For example, hippocampal sequences in a
memory task with rats were found to be predictive of future behaviour
(Pastalkova et al.,, 2008), and sequences in the hippocampus appear to encode
and retain context-specific information (MacDonald et al.,, 2011) (Fig. 1 a).
Sequences of hippocampal place cells encoding trajectories in physical space
were observed during rest and sleep periods after the relevant spatial
experience (Skaggs and McNaughton,, 1996) (see Fig. 1 b), and similar
observations were made during rest before the task (Dragoi and Tonegawa,,
2011).

In other areas of the neurosciences, a mesmerising collection of findings has
been published in recent years, all of which point at sequential neuronal
activity underlying a wide range of distinct perceptual and cognitive
processes: For example, sequential activation patterns in working memory tasks
have been observed in a number of different brain regions, e.g. the parietal
(Harvey et al.,, 2012) and prefrontal cortex (Baeg et al.,, 2003). Neuronal
sequence detectors were identified in the somatosensory cortex (Laboy-Juárez
et al.,, 2019), and sequential activation of neurons in the gustatory cortex
of rats were observed to be involved in the representation of taste (Jones et
al.,, 2007) (Fig. 1 d). There is also evidence for spontaneous neuronal
sequences in the visual cortex (Kenet et al.,, 2003). In songbirds, temporally
highly precise sequences of neuronal firing have been observed during song
production and sleep in the so-called HVC and RA regions of the bird brain
(Hahnloser et al.,, 2002) (Fig. 1 c). In a delayed localisation task, neurons
in the frontal cortex of monkeys were found to go through a series of discrete
states in a sequence which was predictive of the monkeys’ responses (Seidemann
et al.,, 1996). Recently, spike sequences in the amygdala were found to encode
valence-related information during affective learning (Reitich-Stolero and
Paz,, 2019). Even at the cellular level, there is evidence of sequence
processing capacities of single neurons (Branco et al.,, 2010).

These are a few examples but given the apparent ubiquity of sequential
neuronal activity, the question about the potential functional role of
neuronal sequences arises. Is there some underlying principle which could
unify the functional interpretation of sequential activity in different brain
areas? One intuitive explanation is that sequential activity may form a basis
for the temporal representation of past, present and future, and is therefore
useful to encode the spatio-temporal structure of sensory stimuli (Friston and
Buzsáki,, 2016; Buonomano and Maass,, 2009). In particular, theoretical
frameworks like predictive coding and the Bayesian brain hypothesis posit that
one main function of the brain is to predict future states (Friston and
Kiebel,, 2009; Knill and Pouget,, 2004; Rao and Ballard,, 1999). Consequently,
one intriguing idea is that the brain’s internal representation and especially
prediction relies on sequential representations implemented by neuronal
activity (FitzGerald et al.,, 2017; Kiebel et al.,, 2009; Hawkins et al.,,
2009). To infer the hidden causes of sensations, generate predictions and
adaptive motor responses, the brain should use a sequential structure similar
to the causal structure of the sensory input.

Assuming that sequences are a good basis for entertaining predictions about
what happens next in our environment, the next question is what sequential
computational mechanism the brain uses to makes sense of the world and
generate predictions and actions.

To address this question, we will give a short introduction to the core
concepts underlying the brain’s function as a predictive organ. This is
followed by a discussion on how sequences are typically modelled in the
neurosciences and how sequences and hierarchies of sequences are used in both
the neurosciences and machine learning. We conclude with a brief outlook on
how the principles described here may be useful for converging research in the
neurosciences and machine learning.

Figure 1: Four illustrative examples of sequential neuronal activity in
different paradigms and experimental contexts. a Neuronal sequences in the rat
hippocampus during different phases of a behavioural experiment. Each row
shows the raster plots and peristimulus spike histograms of a different
neuron. During the experiment, cells fired sequentially and responded
differentially to GO trials (green) and NoGO trials (red). Reprinted from
(MacDonald et al.,, 2011) with permission from Elsevier. b Sequential
activation of rat hippocampal cells are not only found during the relevant
action but also during rest phases after the behavioural tasks. The top plot
shows the spiking histogram of 91 hippocampal cells during a rat’s trip along
a physical track. The bottom panel shows the rat’s actual position on the
track (blue line) against the position inferred from the spiking pattern of
its hippocampal cells. After the traversal of the track, hippocampal cells
“replayed” their activation sequence in reverse during a short ripple (red
box, enlarged in the box on the right). Figure adapted with permission from
(Pfeiffer,, 2020) (Copyright 1999-2019 John Wiley & Sons, Inc) under the
Creative Commons License. (DOI). c Zebra finches are songbirds whose songs
consist of highly consistent so-called song motifs. Here, the activation of
three different neurons in the RA nucleus of the zebra finch brain during two
renditions of the same song motifs is shown. Adapted by permission from
Springer Nature (Hahnloser et al.,, 2002). d Firing patterns of neurons in the
gustatory cortex of rats in vivo when presented with four different odours.
While the firing of any single neuron was not predictive of the odour, the
sequential switching of states of a Hidden Markov Model (HMM, see section 3.1)
was determined by the presented aroma. For each of the four odours, the
different colour hues represent different HMM states that were inferred based
on the data. Adapted with permission from (Jones et al.,, 2007) (Copyright
(2007) National Academy of Sciences, U.S.A.).

###  1.2 The predictive brain

Until the early 2000s, the main body of neuroscience research focused on
specific functions and faculties of the brain, e.g., memory, attention and
visual perception. The prevailing aim was to find out how the brain
implemented these functions. This was followed by a ’predictive revolution’ in
cognitive neuroscience, where key publications like (Friston,, 2005; Rao and
Ballard,, 1999; Knill and Pouget,, 2004) would inspire cognitive
neuroscientists to integrate the obvious need of the brain to make predictions
with specific research questions, e.g. (Bubic et al.,, 2010; Büchel et al.,,
2014).

Importantly, accurate predictions require continuous updating of the causal
model of sensory stimuli. Combining the Bayesian brain hypothesis with
predictive coding provides a theoretical basis for computational mechanisms
that drive a lifelong learning of the hierarchical causal model of the world
(Friston et al., 2014b, ). Under the Bayesian view the neuronal activity does
not only encode predictions (and prediction errors) about the hidden causes of
sensory stimuli, but additionally encodes a measure of uncertainty about that
information (Knill and Pouget,, 2004). In order to infer the hidden states of
its environment, i.e. to perform recognition, an agent evaluates previously
acquired information and incoming sensory data on the basis of an internal
generative model of itself and its surroundings. On the basis of this model,
the agent’s experience is consolidated into a prior distribution, or prior
belief, and incoming sensory information informs the so-called likelihood.
Bayesian inference describes how prior and likelihood are combined in an
optimal manner to create the posterior distribution, which is the agent’s
belief about the hidden states of the world (Knill and Pouget,, 2004).

Implementations of the Bayesian brain hypothesis have been used to explain,
among other things, movement planning (Friston et al.,, 2015) and the
computational origins of brain dysfunctions such as psychosis (Adams et al.,,
2013; Corlett et al.,, 2009), and schizophrenia (Fletcher and Frith,, 2009).
See also (Clark,, 2013) for a review. Importantly, the Bayesian brain
hypothesis (Doya et al.,, 2007) receives experimental support by a wide range
of behavioural findings which show that humans tend to act Bayes-optimally, as
in, for example, multi-sensory perception (Rohe and Noppeney,, 2015; Kayser
and Shams,, 2015), spatial and temporal perception (O’Reilly et al.,, 2013;
Shi et al.,, 2013), adaptive learning (Soltani and Izquierdo,, 2019; Moens and
Zénon,, 2019), perceptual and goal-directed decision making (Friston et al.,
2014a, ; Bitzer et al.,, 2014; Meyniel et al.,, 2015; Poudel et al.,, 2017).

The quality of the inference, that is, the quality of the belief about the
hidden states of the world, is dependent on the quality of the agent”s
generative model, and the appropriateness of a tractable (approximate)
inference scheme. In this review paper, we suggest that one hallmark of these
models in our typical environment is their sequential structure, and that such
a sequential structure can be used to robustly perform tractable inference on
sequentially structured data.

###  1.3 What are sequences?

Historically, neuroscience research was focused less on the temporal domain
but on more obvious, accessible research questions, like the brain’s
representation of space (Hubel and Wiesel,, 1959; O’Keefe and Dostrovsky,,
1971). This focus on non-temporal research questions was probably supported,
in particular before the predictive revolution, by an experimental view that a
stimulus is an impulse which causes measurable short-lived neuronal
perturbations. In this view, such perturbations are analysed and interpreted
as a system response without considering the possibility that the response may
be part of some underlying, ongoing dynamics. More recently, it became more
evident that spontaneous brain activity may be crucial for understanding brain
function (Fox et al.,, 2005; Deco et al.,, 2011; Kenet et al.,, 2003).

What does it mean to refer to neuronal activity as sequential? In the most
common sense of the word, a sequence is usually understood as the serial
succession of discrete elements or states. Likewise, when thinking of
sequences, most people intuitively think of examples like ’A, B, C, …’ or ’1,
2, 3, …’. However, when extending this discrete concept to neuronal sequences,
there are only few compelling examples where spike activity is readily
interpretable as a discrete sequence, like the ’domino-chain’ activation
observed in the birdbrain nucleus HVC (Hahnloser et al.,, 2002). However, it
is reasonable to assume that sequential representations are used in many brain
areas, especially those directly connected to the hippocampus. Hippocampal-
cortical interactions are characterised by so-called sharp wave ripples
(Buzsáki,, 2015), which are effectively compressed spike sequences. Recent
findings suggest that other cortical areas connected to the hippocampus use
grid-cell like representations similar to space representation in the
hippocampus (Constantinescu et al.,, 2016; Stachenfeld et al.,, 2017). This
suggests that at least areas connected to the hippocampus are able to decode
spike sequences.

To close the gap between, on one side, the seemingly continuous dynamics in
our environment, our sensory input and our own movements and, on the other
side, discrete sequences, one can use so-called winnerless competition (WLC)
dynamics (Rabinovich et al.,, 2000; Afraimovich et al., 2004b, ; Rabinovich et
al.,, 2008). WLC dynamics have been used to characterise sequential neuronal
activity as a succession of discrete-like states which serve as waypoints for
the continuous evolution of neuronal dynamics and cognitive states. In this
paper we introduce the similar concept of continuodiscrete trajectories, which
are characterised by concrete states (fixed points) which are connected by
continuous trajectories. Continuodiscrete trajectories can exhibit WLC
dynamics, while they can also remain in one state indefinitely. The concept of
continuodiscrete dynamics is schematically illustrated in Figure 2. Such an
approach makes intuitive sense for describing behaviour, as many movement
sequences are structured in this fashion. One illustrative example is dancing,
which can be described as a succession of postures and steps, which serve the
function of discrete waypoints for the choreography. These discrete sequence
elements are then “connected” by dynamic body movements. Another intuitive
example is speech, which can, at a high level, be modelled as a discrete
sequence of words. At lower levels, speech can be expressed as continuous
dynamics (vocal tract movements) between specific postures of the vocal tract
(discrete waypoints), see e.g. (Birkholz et al.,, 2010).

Figure 2: Illustration of hierarchical continuodiscrete dynamics. Slowly
changing dynamics at the top level parametrize the sequence of states of the
faster changing second-level dynamics $z^{(2)}$. As the dynamics of variables
$\vec{x}^{(i)}$ iterates between the binary states “on” and “off”, their
behaviour constitutes continuodiscrete WLC dynamics. At around iteration step
$600$, the green unit at the third level (element $\vec{x}^{(3)}_{3}$) becomes
active, which changes the second-level sequential dynamics from
red$\rightarrow$green$\rightarrow$orange$\rightarrow$blue$\rightarrow$red to
red$\rightarrow$blue$\rightarrow$green$\rightarrow$orange$\rightarrow$red.
This is achieved by parametrization of the second-level connectivity matrix
$\rho^{(2)}$ based on the activity on the third level. The second-level
dynamics in turn parametrize an Ornstein-Uhlenbeck process, where the activity
of the second-level units drives the process towards the attractors specified
in vector $\boldsymbol{b}=(0.6,0,-1,-0.3)^{T}$. The rate of change per unit
time is parametrized by vector $\boldsymbol{a}=(1,0.5,1.2,0.8)^{T}$. $\sigma$
is the softmax function which is applied elementwise. 1 denotes a vector of
ones. $\kappa=2$, $\lambda=1/8$. Grey vertical lines in the first level mark
the moment of change between states in the second level. This hierarchical
parametrization of sequences is similar to the approach in (Kiebel et al.,,
2009).

For the brain, representing continuodiscrete trajectories seems to combine the
best of two worlds: Firstly, the representation of discrete-like waypoints
forms the basis for the generalisation and categorisation of the sequence. For
example, for the categorisation of a specific movement sequence, it is not
necessary to consider all the details of the sensory input, as it is
sufficient to categorise the sequence type (dancing, walking, running) by
recognising the sequence of discrete-like waypoints, as e.g. in (Giese and
Poggio,, 2003). Secondly, the brain requires a way of representing continuous
dynamics to not miss important details. This is because key information can
only be inferred by subtle variations within a sequence, as is often the case
in our environment. For instance, when someone is talking to us, most of the
speech content is represented by discrete-like waypoints that describe vocal
tract movements, and body movements like gestures. Additionally however, there
are subtle variations in the exact expression of the discrete-like waypoints
which let us infer about otherwise hidden states like the emotional state of
the speaker (Birkholz et al.,, 2010; Kotz et al.,, 2003; Schmidt et al.,,
2006). In other words, representing sensory input as continuodiscrete
trajectories enables the recognition of invariances of the underlying
movements without losing fine-grained details.

There is growing evidence that sequences with discrete-like states are a
fundamental feature of cognitive and perceptual representations, e.g.
(Seidemann et al.,, 1996; Jones et al.,, 2007). This feature may be at the
heart of several findings in the cognitive sciences which suggest that human
perception is chunked into discrete states, see (VanRullen and Koch,, 2003)
for some insightful examples. Assuming that the brain uses some form of
continuodiscrete trajectories to represent sensory input, the next question is
how this can be expressed mathematically.

##  2 Artificial Neural Networks as Sequence Generators

Artificial Neural Networks generally fall into one of two categories: Feed-
forward networks and recurrent networks. The distinguishing difference between
these two approaches is their connectivity architecture. In recurrent neural
networks (RNNs), neurons (units) maintain connections back to themselves,
either directly or indirectly via other neurons. Consequently, a unit’s
activity in a recurrent network has an effect on its activity at some later
time point, even in the absence of time-dependent parameters like refractory
time. This introduces an intrinsic time-dependency, which enables recurrent
networks to deal well with temporally structured data. Conversely, activity of
a unit in a feed-forward network affects only subsequent units, and does not
influence its own activity at some later point, unless this is specifically
modelled with time-dependent parameters. Therefore, it is reasonable to assume
that feed-forward networks are generally more suitable for problems that do
not exhibit an explicit temporal nature, like image recognition. Hence feed-
forward networks may in general not be adequate for sequence generation,
unless they are explicitly modelled to do so (see next section for a prominent
example), while recurrent models of neuronal activity are either sequence
generators by design, or can easily be used for sequence generation
(Breakspear,, 2017; Long and Fang,, 2010; Gros,, 2009). In what follows, we
will briefly describe various dynamical principles implemented in artificial
feed-forward and recurrent neural networks that have been proposed to model
sequential neuronal activity. For RNNs, we will furthermore discriminate
between network architectures which generate random sequences implicitly as a
side-effect of their recurrent connectivities, and architectures which are
explicitly designed to generate a sequential activation of their constituents.

###  2.1 Feed-Forward Neural Networks

A specific arrangement of neurons in a feed-forward chain can be exploited for
sequence generation. Such arrangements, called synfire chains, have been
argued to be possibly the basis of some neuroscientific observations of
sequential activity (Abeles,, 1991; Diesmann et al.,, 1999; Ikegaya et al.,,
2004).

####  2.1.1 Synfire Chains

Figure 3: a Illustration of a synfire chain. Arrows indicate excitatory
connections. b Spiking histogram of neurons in a synfire chain with ten nodes
of 100 neurons each. The average time interval between the firing of two
adjacent groups corresponds to one synaptic delay.

Synfire chains are concatenated groups of excitatory neurons with convergent-
divergent feed-forward connectivity, as illustrated in Fig. 3 (a) (Abeles,,
1991; Diesmann et al.,, 1999). Synchronous activation of one group leads to
the activation of the subsequent group in the chain after one synaptic delay.
It has been shown that the only stable operating mode in synfire chains is the
synchronous mode where all neurons of a group spike in synchrony (Litvak et
al.,, 2003). Synfire chains create a temporally highly precise sequence of
firing neuronal clusters (Abeles,, 1991; Diesmann et al.,, 1999). Such
temporally precise sequences have been observed in slices of the mouse primary
visual cortex and in V1 of anaesthetised cats (Ikegaya et al.,, 2004), as well
as in the HVC song nucleus of the bird brain during song production (Hahnloser
et al.,, 2002; Long et al.,, 2010). Synfire chains might be one explanation
for these observations. A striking mismatch to neuronal networks in the brain
however is the absence of recurrent connections in the synfire chain’s feed-
forward architecture. Modelling studies have shown that sequential activation
similar to synfire chain activity can be achieved by changing a small fraction
of the connectivities in a random neural network (Rajan et al.,, 2016; Chenkov
et al.,, 2017). This was used to implement working memory (Rajan et al.,,
2016) or give a possible explanation for the occurrence of memory replay after
one-shot learning (Chenkov et al.,, 2017). Such internally generated sequences
have been proposed as a mechanism for memory consolidation, among other things
(see (Pezzulo et al.,, 2014) for a review).

###  2.2 Recurrent Neural Networks

Recurrent Neural Networks (RNN) are an important part of neuroscience, as most
brain regions, and especially the cortex, are recurrently connected
internally. The current role of RNNs in research is diverse, and theoretical
neuroscience employs them in various different approaches. RNNs have been used
to replicate findings from behavioural and imaging studies (Enel et al.,,
2016; Rajan et al.,, 2016; Laje and Buonomano,, 2013; Kim et al.,, 2019), and
some interesting research is dedicated to the evolution of neurobiologically
sound network learning rules (Miconi,, 2017; Memmesheimer et al.,, 2014).
Often, such learning rules are either biologically more plausible versions of
algorithms already known from machine learning, or their actual occurrence in
biological systems remains yet to be shown. Another stream of research
investigates the capabilities and characteristics of networks which learn
based on biologically well-established concepts like Hebbian plasticity and
spike-time dependent plasticity (Sprekeler et al.,, 2007; Legenstein et al.,,
2010; Chenkov et al.,, 2017). A review of the current state of learning in
biologically plausible spiking neural networks can be found in (Taherkhani et
al.,, 2020).

####  2.2.1 Limit Cycles

Figure 4: Two different representations of limit cycles. a A Limit cycle in
three-dimensional phase space. The dimensions of the phase space can be
interpreted as the firing rate of single units. b Representation of a six-
dimensional limit cycle as alternating activations of six different units.

Limit cycles are important objects in dynamic systems theory, and they occur
practically in every physical domain (Strogatz,, 2018). While limit cycles are
continuous trajectories in phase space and do therefore not fall within the
realm of continuodiscrete dynamics, experimentally they can be barely
distinguishable from continuodiscrete trajectories of high temporal precision.
We will therefore briefly discuss limit cycles and their application in
current neuroscience research.

A limit cycle is a closed trajectory, with fixed period and amplitude, in the
phase-space of a dynamical system (Fig. 4). Nearby trajectories are either
attracted towards or repelled from a limit cycle. Limit cycles occur
frequently in biological and other dynamical systems, e.g. the beating of the
heart, or the periodic firing of a pacemaker neuron are regulated by limit
cycles (Strogatz,, 2018). Here, we are interested in limit cycles in the phase
space of a neuronal network. Limit cycles can occur in nonlinear, i.e.
recurrent, neural networks (Berry and Quoy,, 2006; Jouffroy,, 2007). Limit
cycles by definition have smooth, continuous phase-space trajectories. They
are of great interest to theoretical neuroscience, as periodic spiking
activity can be modelled by limit cycles, both on single-cell level
(Izhikevich,, 2007) and population level (Mi et al.,, 2017). They also play an
important role in the emulation of human motion in robotics, where a study
conducted on human participants reported that there are two different types of
human motion (Schaal et al.,, 2007). The first type is rhythmic motion, like
the pedalling on a bike, or the rhythmic step-by-step alternation of the feet
during walking. The second type is called “motion strokes” which comprises
directed non-rhythmic motion, like reaching for a glass of water. These two
types of motion can be modelled by the unifying approach of dynamic motion
primitives (DMPs) (Schaal et al.,, 2007). The main idea of DMPs is that the
limbs move as if they were pulled towards an attractor state. In the case of
rhythmic motion, the attractor is given by a limit cycle, while in the case of
motion strokes the attractor is a discrete point in space. This suggests that
sequences and limit cycles serve similar purposes.

####  2.2.2 Heteroclinic Trajectories

Figure 5: a Illustration of a heteroclinic trajectory (solid line) in three-
dimensional phase space. Dashed lines indicate invariant manifolds between
saddle states, i.e. heteroclinic connections. The green tube illustrates a
Stable Heteroclinic Channel. All heteroclinic trajectories originating in the
SHC will remain inside of it. b Simulation of an SHC-trajectory based on
Lotka-Volterra dynamics. c Neuronal responses to odour representation in the
locust brain. Figures b and c are adapted with permission from (Rabinovich et
al.,, 2001) (DOI). Copyright (2001) by the American Physical Society.

Another approach to model sequential neuronal dynamics are heteroclinic
networks (Ashwin and Timme,, 2005; Rabinovich et al.,, 2008). A heteroclinic
network is a dynamic system with semi-stable states (saddle points) which are
connected by phase-space trajectories. Symmetric networks of coupled
oscillators exhibit useful mathematical properties, including the occurrence
of robust heteroclinic cycles (Ashwin and Swift,, 1992). This is interesting
for theories of neuronal networks as the activity of single neurons is well
described by oscillator models, and synchronized activity of neurons can be
described by coupled oscillator models (Ashwin et al.,, 2007). Coupled
oscillator networks were observed to exhibit heteroclinic behaviour. This
behaviour is characterised by phase space trajectories along the heteroclinic
connections of the network (Fig. 5). This itineration can be guided by
external input, where the travelled trajectory discriminates between different
inputs (Ashwin et al.,, 2007). It was furthermore shown that such heteroclinic
networks generally possess the ability to perform computational tasks (Ashwin
and Borresen,, 2005), and that it is possible to implement any logic operation
within such a network (Neves and Timme,, 2012). Closely related to the dynamic
alteration between states in a heteroclinic network is the previously
introduced concept of winnerless competition (WLC) (Rabinovich et al.,, 2000;
Afraimovich et al., 2004b, ; Rabinovich et al.,, 2008) (see Sec. 1.3). This
concept can be viewed as a counterpoint to the well-known winner-take-all
dynamics of sensory coding, as is for example observed in Hopfield networks
(Hopfield,, 1982). WLC was proposed as a general information processing
principle for dynamical networks and is characterised by dynamic switching
between network states, where the switching behaviour is based on external
input (Afraimovich et al., 2004b, ). Importantly, the travelled trajectory
identifies the received input, while any single state of the trajectory
generally does not, see for example (Neves and Timme,, 2012). In phase space
representation, WLC can be achieved by open or closed sequences of
heteroclinically concatenated saddle points. Such sequences are termed stable
heteroclinic sequences (SHS) if the heteroclinic connections are dissipative,
i.e. when a trajectory starting in a neighbourhood close to the sequence
remains close (Afraimovich et al., 2004a, ). The application of SHS to real
physical dynamical systems, like networks of biological neurons, is however
limited as many kinds of external and internal perturbations can destroy an
SHS. If trajectories of an SHS are preserved, the sequence and its surrounding
stable neighbourhood are called a Stable Heteroclinic Channel (SHC) (see Fig.
5) (Rabinovich et al.,, 2006). A simple implementation of SHCs is based on the
generalised Lotka-Volterra equations (Bick and Rabinovich,, 2010), which are a
type of recurrent neural network implicitly implementing the WLC concept. The
temporal precision of a system that evolves along an SHC is defined by the
noise level as well as the eigenvalues of the invariant directions of the
saddle points. Therefore, sequences along heteroclinic trajectories are
reproducible although the exact timing of the sequence elements may be subject
to fluctuation.

In a similar approach, recent theoretical work on the behaviour of RNNs has
introduced the concept of excitable network attractors, which are
characterised by stable states of a system connected by excitable connections
(Ceni et al.,, 2019). The conceptual idea of orbits between fixed points may
further be implemented in different ways. For instance, transient activation
of neuronal clusters can be achieved by introducing bifurcations such that the
currently stable fixed point becomes unstable, thus forcing the system to
approach a new stable state (Gros,, 2007).

##  3 Recognition of sequences

Although neuronal sequence models such as the ones introduced in the preceding
sections have been used to model experimentally observed neuronal data, these
models by themselves do not explain how predictions are formed about the
future trajectory of a sequence. Rather, neuronal sequence models like synfire
chains or SHCs only describe how trajectories travel from one waypoint to
another, in an autonomous fashion. To take the example of song production and
recognition in songbirds, if we have a sequential model of birdsong
generation, e.g. the song of a Northern cardinal, this is not be sufficient to
model or explain how the listening female bird recognises the ongoing song
(Yildiz and Kiebel,, 2011). To model recognition, one needs a predictive model
which compares the listening female’s ongoing expectations of the male’s song
with the auditory input. One can generalise this point to any sensory input:
To recognise and predict the observed sensory input caused by movement of
others (or oneself), one needs a recognition model with expectations about the
expected, underlying movement sequence that caused the sensory input. One way
to do this is to combine sequential models with the Bayesian brain hypothesis.
The idea is to use sequential models as generative models of sensory input so
that one can derive Bayesian recognition models for sequentially generated
sensory input. A simple example of such a scheme is provided in (Bitzer and
Kiebel,, 2012), where RNNs are used as a generative model such that model
inversion provides for an online recognition model. As shown in (Friston et
al.,, 2011), one can also place such a generative model into the active
inference framework to derive a model that not only recognises sequential
movements from visual input but also generates sequential movement patterns.
Such models are not only interesting from a cognitive neuroscience perspective
but also point at a shared interest with the field of artificial intelligence
and specifically machine learning, of better understanding the nature of
generative models employed by the brain. In other words, both cognitive
neuroscience and artificial intelligence strive to find a mechanistic
understanding of how sensory input caused by others can be recognised by an
artificial or a biological agent. In the following, we will review how both
fields seem to converge on the conceptual idea that such models should be
sequential and hierarchical.

###  3.1 Sequence recognition in machine learning

Given that much of our sensory input can be modelled as continuodiscrete
sequences, the question is how an artificial agent can recognise such
sequences under the Bayesian brain hypothesis. Sequential inference methods
have a long history in both machine learning and neuroscience. The most widely
used models for discrete sequence generation are hidden Markov models (HMM)
and their time-dependent generalisation, hidden semi-Markov models (HSMM)
(Yu,, 2015). In particular, HMMs and HSMM are a standard tool in a wide range
of applications concerned with recognising human movements, e.g. in speech
recognition (Liu et al.,, 2018; Zen et al.,, 2004; Deng et al.,, 2006) and
activity recognition (Duong et al.,, 2005).

Similar to HSMMs, artificial RNNs are actively used in machine learning for
classifying and predicting time series data. When training a generic RNN for
prediction and classification of time series data, one faces various
challenges, most notably incorporating information about long-term
dependencies in the data. To address these dependencies, specific RNN
architectures have been proposed, such as long-short term memory (LSTM)
networks (Gers et al.,, 1999) and gate recurrent units (GRU) (Chung et al.,,
2014). In a common LSTM network, additionally to the output variable, the
network computes an internal variable called the memory cell, which it passes
on to itself upon each iteration. This endows the network with high
flexibility. LSTM networks belong to the most successful and most widely
applied RNN architectures, with applications in virtually every field
involving time-series data, or any data structure with long-range dependencies
(Yu et al.,, 2019; LeCun et al.,, 2015). Another RNN approach is reservoir
computing (RC), which started with the development of echo-state networks and
spiking liquid state machines in the early 2000s (Lukoševičius et al.,, 2012;
Jaeger,, 2001; Maass et al.,, 2002). In RC, sequential input is fed to one or
more input neurons. Those neurons are connected with a reservoir of randomly
connected neurons, which in turn are connected to one or more output neurons.
During training of a reservoir computing network only the output weights are
adjusted. RC networks have successfully been applied in almost every field of
machine learning and data science, such as speech recognition, handwriting
recognition, robot motor control and financial forecasting (Lukoševičius et
al.,, 2012; Tanaka et al.,, 2019).

In summary, machine learning approaches for sequence prediction and
classification are developing in the direction of models capable of
representing sequences. The existence of different long-term dependencies does
furthermore suggest that sequences can be present over a mixture of time
scales, see below.

Although machine recognition models with intrinsic sequential dynamics are
commonplace in computer science and AI research, most existing models used in
machine learning are not neurobiologically plausible mainly for two reasons:
either they are purely discrete (HSMM), or use complex non-local learning
schemes (e.g. backpropagation signals over multiple layers), although there is
research on neurobiologically sound learning paradigms for RNNs (Miconi,,
2017; Taherkhani et al.,, 2020). One alternative is to use neurobiologically
plausible sequence generation models, which can act as generative models of
the causal dynamic relationships in the environment. A natural application
would be the development of models of the recognition process in terms of
Bayesian inference (Bitzer and Kiebel,, 2012), and more specifically in terms
of variational inference (Friston et al.,, 2006; Daunizeau et al.,, 2009).

###  3.2 Biological and Artificial Inferential Hierarchies

In neuroscience and the cognitive sciences, the brain is often viewed as a
hierarchical system, where a functional hierarchy can be mapped to the
structural hierarchy of the cortex (Badre,, 2008; Koechlin et al.,, 2003;
Kiebel et al.,, 2008). The best example of such a hierarchical organisation is
the visual system, for which the existence of both a functional and an
equivalent structural hierarchy is most unambiguously established (Felleman
and Van Essen,, 1991). Cells in lower levels of the hierarchy encode simple
features and have smaller receptive fields than cells further up the
hierarchy, which in turn posses larger receptive fields and encode more
complex patterns by integrating information from lower levels (Hubel and
Wiesel,, 1959; Zeki and Shipp,, 1988; Giese and Poggio,, 2003). This
functional hierarchy is mediated by an asymmetry of recurrent connectivity in
the visual stream (Zeki and Shipp,, 1988; Sherman and Guillery,, 1998).
Hierarchical organisation of a biological neuronal system appears furthermore
to occur naturally as a result of connection costs between neural ensembles,
which has been shown in a modelling study with artificial neurons (Mengistu et
al.,, 2016). Hierarchies also play an important role in cognitive neuroscience
as most if not all types of behaviour as well as cognitive processes can be
described in a hierarchical fashion. For example, the action of making a cup
of tea can be considered a high-order goal in a hierarchy with various
subgoals which are temporally less extended and less abstract. In the given
example, these subgoals can be: (i) putting a teabag into a pot, (ii) pouring
hot water into the pot and (iii) pouring tea into a cup (example adopted from
(Botvinick,, 2007)).

Importantly, all theories of cortical hierarchies of function share the common
assumption that primary sensory regions are at the bottom of the hierarchy and
are influenced by higher order association cortices. This principle has been
conceptualized as a ’hierarchy of time scales’ (Kiebel et al.,, 2008; Hasson
et al.,, 2008; Koechlin et al.,, 2003; Badre,, 2008). In a hierarchy of time
scales the lower levels of the hierarchy represent relatively quickly changing
dynamics, which encode the fast features of sensory input. These levels can
therefore be identified with early sensory regions. Levels further up the
hierarchy code for more general characteristics of the environment and inner
cognitive processes, which generally change slowly (Hasson et al.,, 2008;
Koechlin et al.,, 2003; Badre,, 2008).

The assumption that higher order cortical processes encode more general,
slowly varying features is interesting when considering recent advances in the
field of neuronal computation, where it was shown that spike-time dependent
plasticity enables neurons to extract the slow features of a whitened input
signal (Sprekeler et al.,, 2007). This is a crucial step in the machine
learning algorithm of “slow feature analysis”, which extracts the slow
dynamics from an input signal (Wiskott and Sejnowski,, 2002). The hypothesis
of a hierarchy of time scales is furthermore appealing as it relies on the
rather general concept of time as compared to concepts relating to the
abstractness of representations which may be well defined in behavioural
experiments but are less clear in everyday situations. We assume that the
brain aims to mimic the sensory input it receives in order to predict the
dynamic evolution of its environment. Dynamical processes in the environment
do however evolve on different time scales, where quickly changing dynamics
are often “enslaved” to the slower dynamics. For example, the observation of a
running person is governed by the rather slowly changing dynamics of her
general direction, while these slow dynamics are superposed by the quickly
changing dynamics of her limbs, and the evoked excitation of retinal cells are
in turn superposed by yet faster visual fluctuations of visual input. In order
to make sense of the world and its surroundings, an agent therefore needs to
account for dynamics at different time scales, and be able to relate their
respective influences on each other.

Additional experimental support for the hypothesis of a hierarchy of time
scales has been reported in several neuroimaging studies. In a meta-analysis
of six studies recording single-unit activity in the primate brain during
behavioural tasks, the time scale of brain regions was inferred on the basis
of the autocorrelation function of spike counts for each area (Murray et al.,,
2014). The results provide evidence for an organisation into a hierarchy of
time scales, which was consistent with the anatomical hierarchy inferred from
long-range cortical projections. The sensory cortex consistently showed short
time scales, while the parietal association cortex exhibited intermediate, and
medial prefrontal areas and anterior cingulate cortex (ACC) long time scales
(Murray et al.,, 2014). Furthermore, in an fMRI study, the temporal receptive
windows (TRW) of several brain regions in the human brain were investigated
(Hasson et al.,, 2008). The TRW of an area is the time-interval over which the
region “integrates” incoming information, in order to extract meaning over a
specific temporal scale. It was found that regions such as the primary visual
cortex exhibited rather short TRW, while high order regions exhibited
intermediate to long TRW (Hasson et al.,, 2008).

The importance of cognition at different time scales is also illustrated by
various computational modelling studies. In one study, robots were endowed
with a layered neural network whose parameters were let free to evolve over
time to optimise performance during a navigation task (Nolfi,, 2002). After
some time, the robots had evolved neural assemblies with clearly distinct time
scales: one assembly had assumed a quickly changing, short time scale
associated with immediate sensory input while another assembly had adopted a
long time scale, as the navigation task required integration of information
over an extended period of time. Another modelling study showed that robots
with neuronal populations of strongly differing time-constants performed their
tasks significantly better than when endowed only with units of approximately
identical time-constants (Yamashita and Tani,, 2008). In (Botvinick,, 2007) it
was shown that, after learning, a neural network with a structural hierarchy
similar to one proposed for the frontal cortex had organised in such a way
that high level units coded for temporal context while low-level units encoded
fast responses similar to the role assigned to sensory and motor regions in
theories of hierarchical cortical processing.

Similarly, a hierarchically structured RNN combined with a model of songbirds’
vocal tract dynamics was used in (Yildiz and Kiebel,, 2011) to model the
process of birdsong generation and inference in songbirds. The system
consisted of three levels, each of which governed by the sequential dynamics
of an RNN following a limit cycle. The sequential dynamics were then
influenced both by top-down predictions from the higher level, and prediction
errors from the level below. The model was then used for birdsong recognition.
In another study, the same concept was used for recognition of human speech
(Yildiz et al.,, 2013). The resulting inference scheme was able to recognise
spoken words, even under adversarial circumstances like accelerated speech,
since it entails parameters which are adaptable in an online fashion during
the recognition process. The same principle can also be applied to very
different types of input, see (Rivera et al.,, 2015) for an example in insect
olfaction.

Not surprisingly, the importance of hierarchies of time scales is well
established within the machine learning community (Malhotra et al.,, 2015; El
Hihi and Bengio,, 1996). Current state-of-the-art RNN architectures used for
prediction and classification of complex time series data are based on
recurrent network units organised in temporal hierarchies. Notable examples
are the clockwork RNN (Koutnik et al.,, 2014), gated feedback RNN Chung et
al., (2015), hierarchical multi-scale RNN Chung et al., (2016), fast-slow RNN
Mujika et al., (2017), and a higher order RNN architecture Soltani and Jiang,
(2016). These modern RNN architectures have found various applications in
motion classification Yan et al., (2018); Neverova et al., (2016), speech
synthesis (Zhang and Woodland,, 2018; Achanta and Gangashetty,, 2017; Wu and
King,, 2016), recognition (Chan et al.,, 2016), and other related areas Kurata
et al., (2017); Krause et al., (2017); Liu et al., (2015). Importantly, these
applications of hierarchical RNN architectures further confirm the relevance
of hierarchically organised sequence generators for capturing complex dynamics
in our everyday environments.

##  4 Conclusion

In summary, there is a remarkable body of evidence from both experimental and
computational neuroscience that the brain’s causal model of the environment is
structured as a hierarchy of time scales suitable for generating and
recognising complex sequential activation patterns. In addition, a steadily
increasing amount of empirical evidence suggests that the brain integrates
experience and current sensory information to update its causal model of the
world. Here we reviewed various models of sequential neuronal activity and
work demonstrating how robust inference can be performed using such models. We
argue that generative models of sequences can model and predict sequential
processes in real-world sensory data, especially if that sensory data were
generated by human movements. Inference based on these sequential, generative
models in principle leads to robust belief formation and the adaptation to
changing dynamical conditions in the environment.

Currently, both computational neuroscience and machine learning are developing
in parallel, with many intersections, in an effort to understand key
principles of prediction of, and action upon, complex time series. Clearly,
the effort of machine learning research to represent and predict time series
data in various technological domains, and the effort of neuroscientists to
describe the functional role of sequential neuronal activity in the brain, are
tightly linked. Both fields are constantly evolving and influencing each other
with important empirical data and theoretical concepts. In particular, it is
reasonable to assume that the recognition and prediction of human intentions
and how humans employ cognitive control, in an online fashion, based on
observed human movements, will be a converging point for future developments
both in machine learning and cognitive neuroscience.

##  5 Author Contributions

DM and SK both contributed to the conception of the study. SF wrote the
manuscript, with contributions by DM and SK.

##  6 Conflict of Interest Statement

The authors declare no conflicts of professional, personal, or financial
interest.

##  7 Funding Disclosure

Funded by the German Research Foundation (DFG, Deutsche
Forschungsgemeinschaft), SFB 940/2, project A9, and as part of Germany’s
Excellence Strategy – EXC 2050/1 – Project ID 390696704 – Cluster of
Excellence “Centre for Tactile Internet with Human-in-the-Loop” (CeTI) of
Technische Universität Dresden.

## References

  * Abeles, (1991) Abeles, M. (1991).  Corticonics: Neural circuits of the cerebral cortex.  Cambridge University Press.
  * Achanta and Gangashetty, (2017) Achanta, S. and Gangashetty, S. V. (2017).  Deep elman recurrent neural networks for statistical parametric speech synthesis.  Speech Communication, 93:31–42.
  * Adams et al., (2013) Adams, R. A., Stephan, K. E., Brown, H. R., Frith, C. D., and Friston, K. J. (2013).  The computational anatomy of psychosis.  Frontiers in psychiatry, 4:47.
  * (4) Afraimovich, V., Zhigulin, V., and Rabinovich, M. (2004a).  On the origin of reproducible sequential activity in neural circuits.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 14(4):1123–1129.
  * (5) Afraimovich, V. S., Rabinovich, M. I., and Varona, P. (2004b).  Heteroclinic contours in neural ensembles and the winnerless competition principle.  International Journal of Bifurcation and Chaos, 14(04):1195–1208.
  * Ashwin and Borresen, (2005) Ashwin, P. and Borresen, J. (2005).  Discrete computation using a perturbed heteroclinic network.  Physics Letters A, 347(4-6):208–214.
  * Ashwin et al., (2007) Ashwin, P., Orosz, G., Wordsworth, J., and Townley, S. (2007).  Dynamics on networks of cluster states for globally coupled phase oscillators.  SIAM Journal on Applied Dynamical Systems, 6(4):728–758.
  * Ashwin and Swift, (1992) Ashwin, P. and Swift, J. W. (1992).  The dynamics of n weakly coupled identical oscillators.  Journal of Nonlinear Science, 2(1):69–108.
  * Ashwin and Timme, (2005) Ashwin, P. and Timme, M. (2005).  Nonlinear dynamics: when instability makes sense.  Nature, 436(7047):36.
  * Badre, (2008) Badre, D. (2008).  Cognitive control, hierarchy, and the rostro–caudal organization of the frontal lobes.  Trends in cognitive sciences, 12(5):193–200.
  * Baeg et al., (2003) Baeg, E., Kim, Y., Huh, K., Mook-Jung, I., Kim, H., and Jung, M. (2003).  Dynamics of population code for working memory in the prefrontal cortex.  Neuron, 40(1):177–188.
  * Berry and Quoy, (2006) Berry, H. and Quoy, M. (2006).  Structure and dynamics of random recurrent neural networks.  Adaptive Behavior, 14(2):129–137.
  * Bhalla, (2019) Bhalla, U. S. (2019).  Dendrites, deep learning, and sequences in the hippocampus.  Hippocampus, 29(3):239–251.
  * Bick and Rabinovich, (2010) Bick, C. and Rabinovich, M. I. (2010).  On the occurrence of stable heteroclinic channels in lotka–volterra models.  Dynamical Systems, 25(1):97–110.
  * Birkholz et al., (2010) Birkholz, P., Kroger, B. J., and Neuschaefer-Rube, C. (2010).  Model-based reproduction of articulatory trajectories for consonant–vowel sequences.  IEEE Transactions on Audio, Speech, and Language Processing, 19(5):1422–1433.
  * Bitzer and Kiebel, (2012) Bitzer, S. and Kiebel, S. J. (2012).  Recognizing recurrent neural networks (rrnn): Bayesian inference for recurrent neural networks.  Biological cybernetics, 106(4-5):201–217.
  * Bitzer et al., (2014) Bitzer, S., Park, H., Blankenburg, F., and Kiebel, S. J. (2014).  Perceptual decision making: drift-diffusion model is equivalent to a bayesian model.  Frontiers in human neuroscience, 8:102.
  * Botvinick, (2007) Botvinick, M. M. (2007).  Multilevel structure in behaviour and in the brain: a model of fuster’s hierarchy.  Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1485):1615–1626.
  * Branco et al., (2010) Branco, T., Clark, B. A., and Häusser, M. (2010).  Dendritic discrimination of temporal input sequences in cortical neurons.  Science, 329(5999):1671–1675.
  * Breakspear, (2017) Breakspear, M. (2017).  Dynamic models of large-scale brain activity.  Nature neuroscience, 20(3):340.
  * Bubic et al., (2010) Bubic, A., Von Cramon, D. Y., and Schubotz, R. I. (2010).  Prediction, cognition and the brain.  Frontiers in human neuroscience, 4:25.
  * Büchel et al., (2014) Büchel, C., Geuter, S., Sprenger, C., and Eippert, F. (2014).  Placebo analgesia: a predictive coding perspective.  Neuron, 81(6):1223–1239.
  * Buonomano and Maass, (2009) Buonomano, D. V. and Maass, W. (2009).  State-dependent computations: spatiotemporal processing in cortical networks.  Nature Reviews Neuroscience, 10(2):113.
  * Buzsáki, (2015) Buzsáki, G. (2015).  Hippocampal sharp wave-ripple: A cognitive biomarker for episodic memory and planning.  Hippocampus, 25(10):1073–1188.
  * Ceni et al., (2019) Ceni, A., Ashwin, P., and Livi, L. (2019).  Interpreting recurrent neural networks behaviour via excitable network attractors.  Cognitive Computation, pages 1–27.
  * Chan et al., (2016) Chan, W., Jaitly, N., Le, Q., and Vinyals, O. (2016).  Listen, attend and spell: A neural network for large vocabulary conversational speech recognition.  In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4960–4964. IEEE.
  * Chenkov et al., (2017) Chenkov, N., Sprekeler, H., and Kempter, R. (2017).  Memory replay in balanced recurrent networks.  PLoS computational biology, 13(1):e1005359.
  * Chung et al., (2016) Chung, J., Ahn, S., and Bengio, Y. (2016).  Hierarchical multiscale recurrent neural networks.  arXiv preprint arXiv:1609.01704.
  * Chung et al., (2014) Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014).  Empirical evaluation of gated recurrent neural networks on sequence modeling.  arXiv preprint arXiv:1412.3555.
  * Chung et al., (2015) Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2015).  Gated feedback recurrent neural networks.  In International Conference on Machine Learning, pages 2067–2075.
  * Clark, (2013) Clark, A. (2013).  Whatever next? predictive brains, situated agents, and the future of cognitive science.  Behavioral and brain sciences, 36(3):181–204.
  * Constantinescu et al., (2016) Constantinescu, A. O., O’Reilly, J. X., and Behrens, T. E. (2016).  Organizing conceptual knowledge in humans with a gridlike code.  Science, 352(6292):1464–1468.
  * Corlett et al., (2009) Corlett, P. R., Frith, C. D., and Fletcher, P. C. (2009).  From drugs to deprivation: a bayesian framework for understanding models of psychosis.  Psychopharmacology, 206(4):515–530.
  * Daunizeau et al., (2009) Daunizeau, J., Friston, K. J., and Kiebel, S. J. (2009).  Variational bayesian identification and prediction of stochastic nonlinear dynamic causal models.  Physica D: nonlinear phenomena, 238(21):2089–2118.
  * Deco et al., (2011) Deco, G., Jirsa, V. K., and McIntosh, A. R. (2011).  Emerging concepts for the dynamical organization of resting-state activity in the brain.  Nature Reviews Neuroscience, 12(1):43–56.
  * Deng et al., (2006) Deng, L., Yu, D., and Acero, A. (2006).  Structured speech modeling.  IEEE Transactions on Audio, Speech, and Language Processing, 14(5):1492–1504.
  * Diesmann et al., (1999) Diesmann, M., Gewaltig, M.-O., and Aertsen, A. (1999).  Stable propagation of synchronous spiking in cortical neural networks.  Nature, 402(6761):529.
  * Doya et al., (2007) Doya, K., Ishii, S., Pouget, A., and Rao, R. P. (2007).  Bayesian brain: Probabilistic approaches to neural coding.  MIT press.
  * Dragoi and Tonegawa, (2011) Dragoi, G. and Tonegawa, S. (2011).  Preplay of future place cell sequences by hippocampal cellular assemblies.  Nature, 469(7330):397.
  * Duong et al., (2005) Duong, T. V., Bui, H. H., Phung, D. Q., and Venkatesh, S. (2005).  Activity recognition and abnormality detection with the switching hidden semi-markov model.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 838–845. IEEE.
  * El Hihi and Bengio, (1996) El Hihi, S. and Bengio, Y. (1996).  Hierarchical recurrent neural networks for long-term dependencies.  In Advances in neural information processing systems, pages 493–499.
  * Enel et al., (2016) Enel, P., Procyk, E., Quilodran, R., and Dominey, P. F. (2016).  Reservoir computing properties of neural dynamics in prefrontal cortex.  PLoS computational biology, 12(6):e1004967.
  * Felleman and Van Essen, (1991) Felleman, D. J. and Van Essen, D. (1991).  Distributed hierarchical processing in the primate cerebral cortex.  Cerebral cortex (New York, NY: 1991), 1(1):1–47.
  * FitzGerald et al., (2017) FitzGerald, T. H., Hämmerer, D., Friston, K. J., Li, S.-C., and Dolan, R. J. (2017).  Sequential inference as a mode of cognition and its correlates in fronto-parietal and hippocampal brain regions.  PLoS computational biology, 13(5):e1005418.
  * Fletcher and Frith, (2009) Fletcher, P. C. and Frith, C. D. (2009).  Perceiving is believing: a bayesian approach to explaining the positive symptoms of schizophrenia.  Nature Reviews Neuroscience, 10(1):48–58.
  * Fox et al., (2005) Fox, M. D., Snyder, A. Z., Vincent, J. L., Corbetta, M., Van Essen, D. C., and Raichle, M. E. (2005).  The human brain is intrinsically organized into dynamic, anticorrelated functional networks.  Proceedings of the National Academy of Sciences, 102(27):9673–9678.
  * Friston, (2005) Friston, K. (2005).  A theory of cortical responses.  Philosophical transactions of the Royal Society B: Biological sciences, 360(1456):815–836.
  * Friston and Buzsáki, (2016) Friston, K. and Buzsáki, G. (2016).  The functional anatomy of time: what and when in the brain.  Trends in cognitive sciences, 20(7):500–511.
  * Friston and Kiebel, (2009) Friston, K. and Kiebel, S. (2009).  Predictive coding under the free-energy principle.  Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1521):1211–1221.
  * Friston et al., (2006) Friston, K., Kilner, J., and Harrison, L. (2006).  A free energy principle for the brain.  Journal of Physiology-Paris, 100(1-3):70–87.
  * Friston et al., (2011) Friston, K., Mattout, J., and Kilner, J. (2011).  Action understanding and active inference.  Biological cybernetics, 104(1-2):137–160.
  * Friston et al., (2015) Friston, K., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T., and Pezzulo, G. (2015).  Active inference and epistemic value.  Cognitive neuroscience, 6(4):187–214.
  * (53) Friston, K., Schwartenbeck, P., FitzGerald, T., Moutoussis, M., Behrens, T., and Dolan, R. J. (2014a).  The anatomy of choice: dopamine and decision-making.  Philosophical Transactions of the Royal Society B: Biological Sciences, 369(1655):20130481.
  * (54) Friston, K. J., Stephan, K. E., Montague, R., and Dolan, R. J. (2014b).  Computational psychiatry: the brain as a phantastic organ.  The Lancet Psychiatry, 1(2):148–158.
  * Gers et al., (1999) Gers, F. A., Schmidhuber, J., and Cummins, F. (1999).  Learning to forget: Continual prediction with lstm.
  * Giese and Poggio, (2003) Giese, M. A. and Poggio, T. (2003).  Cognitive neuroscience: neural mechanisms for the recognition of biological movements.  Nature Reviews Neuroscience, 4(3):179.
  * Gros, (2007) Gros, C. (2007).  Neural networks with transient state dynamics.  New Journal of Physics, 9(4):109.
  * Gros, (2009) Gros, C. (2009).  Cognitive computation with autonomously active neural networks: an emerging field.  Cognitive Computation, 1(1):77–90.
  * Hahnloser et al., (2002) Hahnloser, R. H., Kozhevnikov, A. A., and Fee, M. S. (2002).  An ultra-sparse code underliesthe generation of neural sequences in a songbird.  Nature, 419(6902):65.
  * Harvey et al., (2012) Harvey, C. D., Coen, P., and Tank, D. W. (2012).  Choice-specific sequences in parietal cortex during a virtual-navigation decision task.  Nature, 484(7392):62.
  * Hasson et al., (2008) Hasson, U., Yang, E., Vallines, I., Heeger, D. J., and Rubin, N. (2008).  A hierarchy of temporal receptive windows in human cortex.  Journal of Neuroscience, 28(10):2539–2550.
  * Hawkins et al., (2009) Hawkins, J., George, D., and Niemasik, J. (2009).  Sequence memory for prediction, inference and behaviour.  Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1521):1203–1209.
  * Hopfield, (1982) Hopfield, J. J. (1982).  Neural networks and physical systems with emergent collective computational abilities.  Proceedings of the national academy of sciences, 79(8):2554–2558.
  * Hubel and Wiesel, (1959) Hubel, D. H. and Wiesel, T. N. (1959).  Receptive fields of single neurones in the cat’s striate cortex.  The Journal of physiology, 148(3):574–591.
  * Ikegaya et al., (2004) Ikegaya, Y., Aaron, G., Cossart, R., Aronov, D., Lampl, I., Ferster, D., and Yuste, R. (2004).  Synfire chains and cortical songs: temporal modules of cortical activity.  Science, 304(5670):559–564.
  * Izhikevich, (2007) Izhikevich, E. M. (2007).  Dynamical systems in neuroscience.  MIT press.
  * Jaeger, (2001) Jaeger, H. (2001).  The “echo state” approach to analysing and training recurrent neural networks-with an erratum note.  Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148(34):13.
  * Jones et al., (2007) Jones, L. M., Fontanini, A., Sadacca, B. F., Miller, P., and Katz, D. B. (2007).  Natural stimuli evoke dynamic sequences of states in sensory cortical ensembles.  Proceedings of the National Academy of Sciences, 104(47):18772–18777.
  * Jouffroy, (2007) Jouffroy, G. (2007).  Design of simple limit cycles with recurrent neural networks for oscillatory control.  In Sixth International Conference on Machine Learning and Applications (ICMLA 2007), pages 50–55. IEEE.
  * Kayser and Shams, (2015) Kayser, C. and Shams, L. (2015).  Multisensory causal inference in the brain.  PLoS biology, 13(2).
  * Kenet et al., (2003) Kenet, T., Bibitchkov, D., Tsodyks, M., Grinvald, A., and Arieli, A. (2003).  Spontaneously emerging cortical representations of visual attributes.  Nature, 425(6961):954.
  * Kiebel et al., (2008) Kiebel, S. J., Daunizeau, J., and Friston, K. J. (2008).  A hierarchy of time-scales and the brain.  PLoS computational biology, 4(11):e1000209.
  * Kiebel et al., (2009) Kiebel, S. J., Von Kriegstein, K., Daunizeau, J., and Friston, K. J. (2009).  Recognizing sequences of sequences.  PLoS computational biology, 5(8):e1000464.
  * Kim et al., (2019) Kim, R., Li, Y., and Sejnowski, T. J. (2019).  Simple framework for constructing functional spiking recurrent neural networks.  bioRxiv, page 579706.
  * Knill and Pouget, (2004) Knill, D. C. and Pouget, A. (2004).  The bayesian brain: the role of uncertainty in neural coding and computation.  TRENDS in Neurosciences, 27(12):712–719.
  * Koechlin et al., (2003) Koechlin, E., Ody, C., and Kouneiher, F. (2003).  The architecture of cognitive control in the human prefrontal cortex.  Science, 302(5648):1181–1185.
  * Kotz et al., (2003) Kotz, S. A., Meyer, M., Alter, K., Besson, M., von Cramon, D. Y., and Friederici, A. D. (2003).  On the lateralization of emotional prosody: an event-related functional mr investigation.  Brain and language, 86(3):366–376.
  * Koutnik et al., (2014) Koutnik, J., Greff, K., Gomez, F., and Schmidhuber, J. (2014).  A clockwork rnn.  arXiv preprint arXiv:1402.3511.
  * Krause et al., (2017) Krause, J., Johnson, J., Krishna, R., and Fei-Fei, L. (2017).  A hierarchical approach for generating descriptive image paragraphs.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 317–325.
  * Kurata et al., (2017) Kurata, G., Ramabhadran, B., Saon, G., and Sethy, A. (2017).  Language modeling with highway lstm.  In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244–251. IEEE.
  * Laboy-Juárez et al., (2019) Laboy-Juárez, K. J., Langberg, T., Ahn, S., and Feldman, D. E. (2019).  Elementary motion sequence detectors in whisker somatosensory cortex.  Nature neuroscience, pages 1–12.
  * Laje and Buonomano, (2013) Laje, R. and Buonomano, D. V. (2013).  Robust timing and motor patterns by taming chaos in recurrent neural networks.  Nature neuroscience, 16(7):925.
  * LeCun et al., (2015) LeCun, Y., Bengio, Y., and Hinton, G. (2015).  Deep learning.  nature, 521(7553):436–444.
  * Legenstein et al., (2010) Legenstein, R., Chase, S. M., Schwartz, A. B., and Maass, W. (2010).  A reward-modulated hebbian learning rule can explain experimentally observed network reorganization in a brain control task.  Journal of Neuroscience, 30(25):8400–8410.
  * Litvak et al., (2003) Litvak, V., Sompolinsky, H., Segev, I., and Abeles, M. (2003).  On the transmission of rate code in long feedforward networks with excitatory–inhibitory balance.  Journal of neuroscience, 23(7):3006–3015.
  * Liu et al., (2018) Liu, H., He, L., Bai, H., Dai, B., Bai, K., and Xu, Z. (2018).  Structured inference for recurrent hidden semi-markov model.  In IJCAI, pages 2447–2453.
  * Liu et al., (2015) Liu, P., Qiu, X., Chen, X., Wu, S., and Huang, X. (2015).  Multi-timescale long short-term memory neural network for modelling sentences and documents.  In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2326–2335.
  * Long and Fang, (2010) Long, L. and Fang, G. (2010).  A review of biologically plausible neuron models for spiking neural networks.  In AIAA Infotech@ Aerospace 2010, page 3540.
  * Long et al., (2010) Long, M. A., Jin, D. Z., and Fee, M. S. (2010).  Support for a synaptic chain model of neuronal sequence generation.  Nature, 468(7322):394.
  * Lukoševičius et al., (2012) Lukoševičius, M., Jaeger, H., and Schrauwen, B. (2012).  Reservoir computing trends.  KI-Künstliche Intelligenz, 26(4):365–371.
  * Maass et al., (2002) Maass, W., Natschläger, T., and Markram, H. (2002).  Real-time computing without stable states: A new framework for neural computation based on perturbations.  Neural computation, 14(11):2531–2560.
  * MacDonald et al., (2011) MacDonald, C. J., Lepage, K. Q., Eden, U. T., and Eichenbaum, H. (2011).  Hippocampal “time cells” bridge the gap in memory for discontiguous events.  Neuron, 71(4):737–749.
  * Malhotra et al., (2015) Malhotra, P., Vig, L., Shroff, G., and Agarwal, P. (2015).  Long short term memory networks for anomaly detection in time series.  In Proceedings, page 89. Presses universitaires de Louvain.
  * Memmesheimer et al., (2014) Memmesheimer, R.-M., Rubin, R., Ölveczky, B. P., and Sompolinsky, H. (2014).  Learning precisely timed spikes.  Neuron, 82(4):925–938.
  * Mengistu et al., (2016) Mengistu, H., Huizinga, J., Mouret, J.-B., and Clune, J. (2016).  The evolutionary origins of hierarchy.  PLoS computational biology, 12(6):e1004829.
  * Meyniel et al., (2015) Meyniel, F., Sigman, M., and Mainen, Z. F. (2015).  Confidence as bayesian probability: From neural origins to behavior.  Neuron, 88(1):78–92.
  * Mi et al., (2017) Mi, Y., Katkov, M., and Tsodyks, M. (2017).  Synaptic correlates of working memory capacity.  Neuron, 93(2):323–330.
  * Miconi, (2017) Miconi, T. (2017).  Biologically plausible learning in recurrent neural networks reproduces neural dynamics observed during cognitive tasks.  Elife, 6:e20899.
  * Moens and Zénon, (2019) Moens, V. and Zénon, A. (2019).  Learning and forgetting using reinforced bayesian change detection.  PLoS computational biology, 15(4):e1006713.
  * Mujika et al., (2017) Mujika, A., Meier, F., and Steger, A. (2017).  Fast-slow recurrent neural networks.  In Advances in Neural Information Processing Systems, pages 5915–5924.
  * Murray et al., (2014) Murray, J. D., Bernacchia, A., Freedman, D. J., Romo, R., Wallis, J. D., Cai, X., Padoa-Schioppa, C., Pasternak, T., Seo, H., Lee, D., et al. (2014).  A hierarchy of intrinsic timescales across primate cortex.  Nature neuroscience, 17(12):1661.
  * Neverova et al., (2016) Neverova, N., Wolf, C., Lacey, G., Fridman, L., Chandra, D., Barbello, B., and Taylor, G. (2016).  Learning human identity from motion patterns.  IEEE Access, 4:1810–1820.
  * Neves and Timme, (2012) Neves, F. S. and Timme, M. (2012).  Computation by switching in complex networks of states.  Physical review letters, 109(1):018701.
  * Nolfi, (2002) Nolfi, S. (2002).  Evolving robots able to self-localize in the environment: the importance of viewing cognition as the result of processes occurring at different time-scales.  Connection Science, 14(3):231–244.
  * O’Keefe and Dostrovsky, (1971) O’Keefe, J. and Dostrovsky, J. (1971).  The hippocampus as a spatial map: preliminary evidence from unit activity in the freely-moving rat.  Brain research.
  * O’Reilly et al., (2013) O’Reilly, J. X., Jbabdi, S., Rushworth, M. F., and Behrens, T. E. (2013).  Brain systems for probabilistic and dynamic prediction: computational specificity and integration.  PLoS biology, 11(9):e1001662.
  * Pastalkova et al., (2008) Pastalkova, E., Itskov, V., Amarasingham, A., and Buzsáki, G. (2008).  Internally generated cell assembly sequences in the rat hippocampus.  Science, 321(5894):1322–1327.
  * Pezzulo et al., (2014) Pezzulo, G., van der Meer, M. A., Lansink, C. S., and Pennartz, C. M. (2014).  Internally generated sequences in learning and executing goal-directed behavior.  Trends in cognitive sciences, 18(12):647–657.
  * Pfeiffer, (2020) Pfeiffer, B. E. (2020).  The content of hippocampal “replay”.  Hippocampus, 30(1):6–18.
  * Poudel et al., (2017) Poudel, G. R., Bhattarai, A., Dickinson, D. L., and Drummond, S. (2017).  Neural correlates of decision-making during a bayesian choice task.  NeuroReport, 28(4):193–199.
  * Rabinovich et al., (2008) Rabinovich, M., Huerta, R., and Laurent, G. (2008).  Transient dynamics for neural processing.  Science, 321(5885):48–50.
  * Rabinovich et al., (2000) Rabinovich, M., Huerta, R., Volkovskii, A., Abarbanel, H., Stopfer, M., and Laurent, G. (2000).  Dynamical coding of sensory information with competitive networks.  Journal of Physiology-Paris, 94(5-6):465–471.
  * Rabinovich et al., (2001) Rabinovich, M., Volkovskii, A., Lecanda, P., Huerta, R., Abarbanel, H., and Laurent, G. (2001).  Dynamical encoding by networks of competing neuron groups: winnerless competition.  Physical review letters, 87(6):068102.
  * Rabinovich et al., (2006) Rabinovich, M. I., Huerta, R., Varona, P., and Afraimovich, V. S. (2006).  Generation and reshaping of sequences in neural systems.  Biological cybernetics, 95(6):519.
  * Rajan et al., (2016) Rajan, K., Harvey, C. D., and Tank, D. W. (2016).  Recurrent network models of sequence generation and memory.  Neuron, 90(1):128–142.
  * Rao and Ballard, (1999) Rao, R. P. and Ballard, D. H. (1999).  Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.  Nature neuroscience, 2(1):79.
  * Reitich-Stolero and Paz, (2019) Reitich-Stolero, T. and Paz, R. (2019).  Affective memory rehearsal with temporal sequences in amygdala neurons.  Nature neuroscience, 22(12):2050–2059.
  * Rivera et al., (2015) Rivera, D. C., Bitzer, S., and Kiebel, S. J. (2015).  Modelling odor decoding in the antennal lobe by combining sequential firing rate models with bayesian inference.  PLoS computational biology, 11(10):e1004528.
  * Rohe and Noppeney, (2015) Rohe, T. and Noppeney, U. (2015).  Cortical hierarchies perform bayesian causal inference in multisensory perception.  PLoS Biology, 13(2).
  * Schaal et al., (2007) Schaal, S., Mohajerian, P., and Ijspeert, A. (2007).  Dynamics systems vs. optimal control—a unifying view.  Progress in brain research, 165:425–445.
  * Schmidt et al., (2006) Schmidt, K. L., Ambadar, Z., Cohn, J. F., and Reed, L. I. (2006).  Movement differences between deliberate and spontaneous facial expressions: Zygomaticus major action in smiling.  Journal of Nonverbal Behavior, 30(1):37–52.
  * Seidemann et al., (1996) Seidemann, E., Meilijson, I., Abeles, M., Bergman, H., and Vaadia, E. (1996).  Simultaneously recorded single units in the frontal cortex go through sequences of discrete and stable states in monkeys performing a delayed localization task.  Journal of Neuroscience, 16(2):752–768.
  * Sherman and Guillery, (1998) Sherman, S. M. and Guillery, R. (1998).  On the actions that one nerve cell can have on another: distinguishing “drivers” from “modulators”.  Proceedings of the National Academy of Sciences, 95(12):7121–7126.
  * Shi et al., (2013) Shi, Z., Church, R. M., and Meck, W. H. (2013).  Bayesian optimization of time perception.  Trends in cognitive sciences, 17(11):556–564.
  * Skaggs and McNaughton, (1996) Skaggs, W. E. and McNaughton, B. L. (1996).  Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience.  Science, 271(5257):1870–1873.
  * Soltani and Izquierdo, (2019) Soltani, A. and Izquierdo, A. (2019).  Adaptive learning under expected and unexpected uncertainty.  Nature Reviews Neuroscience, 20(10):635–644.
  * Soltani and Jiang, (2016) Soltani, R. and Jiang, H. (2016).  Higher order recurrent neural networks.  arXiv preprint arXiv:1605.00064.
  * Sprekeler et al., (2007) Sprekeler, H., Michaelis, C., and Wiskott, L. (2007).  Slowness: An objective for spike-timing–dependent plasticity?  PLoS Computational Biology, 3(6):e112.
  * Stachenfeld et al., (2017) Stachenfeld, K. L., Botvinick, M. M., and Gershman, S. J. (2017).  The hippocampus as a predictive map.  Nature neuroscience, 20(11):1643.
  * Strogatz, (2018) Strogatz, S. H. (2018).  Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering.  CRC Press.
  * Taherkhani et al., (2020) Taherkhani, A., Belatreche, A., Li, Y., Cosma, G., Maguire, L. P., and McGinnity, T. M. (2020).  A review of learning in biologically plausible spiking neural networks.  Neural Networks, 122:253–272.
  * Tanaka et al., (2019) Tanaka, G., Yamane, T., Héroux, J. B., Nakane, R., Kanazawa, N., Takeda, S., Numata, H., Nakano, D., and Hirose, A. (2019).  Recent advances in physical reservoir computing: A review.  Neural Networks.
  * VanRullen and Koch, (2003) VanRullen, R. and Koch, C. (2003).  Is perception discrete or continuous?  Trends in cognitive sciences, 7(5):207–213.
  * Wiskott and Sejnowski, (2002) Wiskott, L. and Sejnowski, T. J. (2002).  Slow feature analysis: Unsupervised learning of invariances.  Neural computation, 14(4):715–770.
  * Wu and King, (2016) Wu, Z. and King, S. (2016).  Investigating gated recurrent networks for speech synthesis.  In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5140–5144. IEEE.
  * Yamashita and Tani, (2008) Yamashita, Y. and Tani, J. (2008).  Emergence of functional hierarchy in a multiple timescale neural network model: a humanoid robot experiment.  PLoS computational biology, 4(11):e1000220.
  * Yan et al., (2018) Yan, S., Smith, J. S., Lu, W., and Zhang, B. (2018).  Hierarchical multi-scale attention networks for action recognition.  Signal Processing: Image Communication, 61:73–84.
  * Yildiz and Kiebel, (2011) Yildiz, I. B. and Kiebel, S. J. (2011).  A hierarchical neuronal model for generation and online recognition of birdsongs.  PLoS computational biology, 7(12):e1002303.
  * Yildiz et al., (2013) Yildiz, I. B., von Kriegstein, K., and Kiebel, S. J. (2013).  From birdsong to human speech recognition: Bayesian inference on a hierarchy of nonlinear dynamical systems.  PLoS computational biology, 9(9):e1003219.
  * Yu, (2015) Yu, S.-Z. (2015).  Hidden Semi-Markov models: theory, algorithms and applications.  Morgan Kaufmann.
  * Yu et al., (2019) Yu, Y., Si, X., Hu, C., and Zhang, J. (2019).  A review of recurrent neural networks: Lstm cells and network architectures.  Neural computation, 31(7):1235–1270.
  * Zeki and Shipp, (1988) Zeki, S. and Shipp, S. (1988).  The functional logic of cortical connections.  Nature, 335(6188):311.
  * Zen et al., (2004) Zen, H., Tokuda, K., Masuko, T., Kobayashi, T., and Kitamura, T. (2004).  Hidden semi-markov model based speech synthesis.  In Eighth International Conference on Spoken Language Processing.
  * Zhang and Woodland, (2018) Zhang, C. and Woodland, P. C. (2018).  High order recurrent neural networks for acoustic modelling.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5849–5853. IEEE.