# A Survey on Reservoir Computing and its Interdisciplinary Applications
Beyond Traditional Machine Learning ††thanks: Citation: Heng Zhang and Danilo
Vasconcellos Vargas. DOI:10.1109/ACCESS.2023.3299296

Heng Zhang, Danilo Vasconcellos Vargas
Department of Information Science and Technology
Kyushu University
Fukuoka, Japan
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Reservoir computing (RC), first applied to temporal signal processing, is a
recurrent neural network in which neurons are randomly connected. Once
initialized, the connection strengths remain unchanged. Such a simple
structure turns RC into a non-linear dynamical system that maps low-
dimensional inputs into a high-dimensional space. The model’s rich dynamics,
linear separability, and memory capacity then enable a simple linear readout
to generate adequate responses for various applications. RC spans areas far
beyond machine learning, since it has been shown that the complex dynamics can
be realized in various physical hardware implementations and biological
devices. This yields greater flexibility and shorter computation time.
Moreover, the neuronal responses triggered by the model’s dynamics shed light
on understanding brain mechanisms that also exploit similar dynamical
processes. While the literature on RC is vast and fragmented, here we conduct
a unified review of RC’s recent developments from machine learning to physics,
biology, and neuroscience. We first review the early RC models, and then
survey the state-of-the-art models and their applications. We further
introduce studies on modeling the brain’s mechanisms by RC. Finally, we offer
new perspectives on RC development, including reservoir design, coding
frameworks unification, physical RC implementations, and interaction between
RC, cognitive neuroscience and evolution.

_K_ eywords Reservoir computing $\cdot$ Neural networks $\cdot$ Recurrent
neural networks $\cdot$ Nonlinear dynamical systems $\cdot$ Cognitive
neuroscience

##  1 Introduction

Artificial neural networks (ANNs) attract attention in the fields of
artificial intelligence, neuroscience, computer science, and machine learning.
These ANNs can be mainly divided into two architectures: (1) feed-forward
neural networks (FFNNs) and (2) recurrent neural networks (RNNs) [1].  In the
field of neuroscience, it has been realized that the convergent feed-forward
circuit observed in the cerebral cortex of mammals is a method used to encode
relations, allowing cognitive objects to be represented through multi-layered
feed-forward architectures [2].  In machine learning, training FFNNs is a
process that usually involves the optimization of a highly non-convex problem
using gradient descent based methods to find the optimum. One of the biggest
advantages of FFNNs is their ability to deal with static (non-temporal) data
processing tasks such as image recognition [3], object detection [4] and
semantic segmentation [5]. However, samples are normally independently
processed in FFNNs, making it hard to handle temporally correlated events
without memory.

On the other hand, RNNs are models where neurons are recurrently coupled with
feedback connections. The recurrent connections provide rich non-linear
dynamics and memory, which are essential for temporal data and sequential
processing.  However, RNNs can be challenging to train. This is mainly because
they must deal with vanishing and exploding gradient problems, along with
other problems such as longer training time and the need for careful weight
initialization [6, 7]. Back-propagation-through-time (BPTT) [8] and Long
Short-Term Memory (LSTM) [9] networks are two solutions to some of the
problems mentioned above. However, the learning difficulty still exists.

Figure 1: Paper structure showing an overall picture and future trends of
research in reservoir computing.

Reservoir computing (RC) is a bio-inspired, RNN based computational framework
mainly originated from two independent models in the early 2000s: the Echo
State Network (ESN) [10] and the Liquid State Machine (LSM) [11]. Similar and
related concepts include backpropagation-decorrelation (BPDC) learning
algorithm [12] and the cortico-striatal models realized in prefrontal cortex
[13]. These proposed models were then unified as the RC computational
framework by Verstraeten et, al. [14], and were summarized in two reviews [15,
16].

A typical RC network includes a reservoir and a readout layer. Specifically,
the reservoir is a network with randomly connected neurons. It can generate
complex and high-dimensional transient responses to input.  Such responses are
considered as reservoir’s states, forming unique trajectories for each input.
These high-dimensional reservoir’s states are then processed with a simple
readout layer, generating the output. During RC training, the weight
connections between neurons in the reservoir are usually fixed to their
initial random values (i.e., remain untrained), and only the readout layer is
trained by relatively simple learning algorithms (e.g., linear regression).

RC has several advantages, such as a fast training process, simplicity of
implementation, reduction of computational cost, and no vanishing or exploding
gradient problems, among others. Moreover, they do have memory and, for this
reason, are capable of tackling temporal and sequential problems. These
advantages have attracted increasing interest in RC research and many of its
applications since its conception. However, due to the impressive wide-range
results of gradient descent based neural networks—the deep learning
revolution, RC research receded into a niche for a few years after the 2010s.
Recently, there has been a resurgence of interest in RC as a compelling
biological model of neural networks, driven by its potential for scalability
through physical implementations. Additionally, problems in deep learning such
as adversarial attacks and robustness/adaptiveness issues have also
contributed to an increase in interest in alternative paradigms [17, 18, 19,
20, 21, 22].

###  1.1 Aims

The complex dynamics in the reservoir indicate that RC is not only just a
machine learning framework, but also a concept that highly correlates to
physics, biology, and neuroscience. In 2019, a review in physical RC (PRC)
comprehensively summarizes various physical materials and hardware components
that can be used for PRC implementations [23]. In biology and neuroscience, it
was reported in 2013 that parts of the brain’s mechanisms are similar to those
of reservoirs, including mixed selectivity [24] and neuronal oscillations [2].
Although valuable surveys exist in specific research branches, there is still
a notable gap in the literature regarding understanding the intricate
interplay between RC and its connections to physics, biology, and
neuroscience. The absence of such a comprehensive review hinders the
advancement of our knowledge and the exploration of RC’s potential across
diverse domains. Addressing this need and providing a holistic overview would
significantly contribute to bridging this gap, facilitating further research,
and unlocking new insights into the various aspects of RC.

This survey provides a clear exposition of recent developments in RC. While
the literature on RC is vast and fragmented, we aim to provide a uniform
introduction to RC. We begin in Section 2 by introducing the fundamental
concepts of RC. In Section 3, we explore different network architectures and
optimization techniques that can enhance reservoir performance. Section 4
further highlights the recent trends of RC, facilitated by improved training
schemes. Various physical hardware solutions are also reviewed, covering
electronic, optical, and biophysical approaches, among others. Furthermore, we
demonstrate the wide range of applications of RC in Section 5, including
practical engineering, natural science, and social/data science, etc. These
applications often involve time-dependent data, requiring memory and memory-
related processing. Next, by discussing the cortico-striatal models and
coupled oscillator networks in Section 6, we show that high-dimensional
responses triggered by the reservoir’s dynamics offer insights into brain
mechanisms that also exploit a high-dimensional dynamical process.
Additionally, our contributions to RC reside in Section 7, where we present
fresh perspectives on the development of RC and pinpoint open problems that
require further research.

##  2 Basic Framework of RC

In this section, we first review the historical developments in RC. By tracing
its trajectory from initial theoretical foundations to the present state, our
emphasis will then be on highlighting significant works that have played a
crucial role in shaping the current landscape of RC.

###  2.1 History of RC Developments

The first prototype of reservoir computing. Perhaps the birth of the RC
framework was in the 1980s-90s. During that period, some researchers were
focusing on the characterization of the fast eye movements (i.e., the
oculomotor saccade) in the corticostriatal system—the interactions between
cortex and basal ganglia [25]. In a pioneering work (1989), Barone and Joseph
[26] examined the function of the corticostriatal system by carrying saccade
experiments on macaque monkeys. They found that some neurons have a preferred
spatial saccade amplitude and direction, resulting in selective responses to a
particular sequential order.  This finding was further characterized as mixed
selectivity by [24] in 2013, which is one of the important principles in both
RC and cognitive science (see Section 6).

Taking inspiration from the experiments of the corticostriatal saccade system
[26, 27], Dominey [13, 28] developed the very first prototype of the reservoir
network in 1995 (see Section 6 and Fig. 15 for detail).  Specifically, the
model was built based on a recurrent prefrontal cortex (PFC) system (the
reservoir), and a reward-related learning method in PFC-to-caudate connections
(the readout). Since they found that the modifications of the recurrent
connections are considerably computationally costing, they decided to
initialize the PFC layer with a mixture of fixed inhibitory and excitatory
recurrent connections (i.e., a reservoir with fixed connections between
neurons). The reservoir was then connected to the caudate or striatum to
obtain the readout.

Early research and problems identified in general RNNs. The main branch of RC
was originated in the fields of temporal and sequential pattern recognition
using RNNs. In contrast to FFNNs that aim to approximate non-linear input-
output functions, RNNs are capable of representing dynamical systems and
processing sequential inputs with recurrent connections [23, 15].

Early studies of RNNs include a well-known model called Hopfield network [29]
in the 1980s. The network topologies were specifically formulated with
symmetrical weights connections and were trained in unsupervised ways. This
special type of network normally experiences chaotic or stochastic dynamics
with the mathematical background of statistical physics [15]. Another type of
RNN features a deterministic update dynamics and directed weighted
connections. Systems from this type of RNN are usually made of high
dimensional hidden states with non-linear dynamics, resulting in a
transformation from an input sequence into an output sequence. Two standard
examples are (1) back-propagation-through-time (BPTT) by [8, 30]; and (2)
real-time recurrent learning (RTRL) by [31]. Even though these learning
methods showed great potential in complex sequential processing, they
struggled to tackle real-world problems due to the high computational costs
and difficulties of training, especially the vanishing and exploding gradient
problems that make them hard to capture long-term dependencies [6, 7]. In
1997, a well-known architecture, Long Short Term Memory (LSTM) [9], was then
proposed to address these problems. For more details on the gradient-based
RNNs, please refer to an early review [32].

Unification of reservoir computing. In 2000, [32] proposed a new algorithm
based on error gradient approximation, which efficiently reduces the
computational complexity and shows faster convergence in recurrent network
training. This work, referred to Atiya and Parlos Recurrent Learning (APRL) in
later literature, identified that an RNN can be divided into two parts: the
quickly changing output weights, and the slowly adapting hidden weights.
Therefore, APRL is considered the algorithm to bridge between general RNNs and
reservoir computing [15]. Besides, another predecessor of RC, the
backpropagation-decorrelation (BPDC) learning algorithm, further simplified
APRL and made it an online learning algorithm [12].

Later in the early 2000s, two types of fundamental reservoir computing
algorithms were independently invented by Maass et al. [11] as Liquid State
Machine (LSM), and by Jaeger [10] as Echo State Network (ESN). The two
algorithms, as well as other related works such as BPDC and works in
neuroscience fields such as Dominey’s research [13], were unified as a
computational framework called reservoir computing (RC) [14]. In this unified
framework, the low-dimensional input data is transformed into spatio-temporal
patterns in a high-dimensional space by the reservoir—an RNN with fixed
topologies and unchanged weights. The high-dimensional responses generated by
the reservoir are then processed by the readout—an output layer which can be
trained with simple learning algorithms such as linear regression. In other
words, during training, the values of the weight connections within the
reservoir remain unchanged while only the readout weights are trained based on
specific tasks. Before going deep into recent advances in RC, we will first
introduce the basic concepts of ESN and LSM in the following subsections.

###  2.2 Echo State Networks

Figure 2: Simplified structure of Echo State Network (ESN).

Echo State Network (ESN) was first proposed by [10]. This pioneering work is
based on the fact that training only the readout layer of an RNN can achieve
acceptable performance, if the network has sufficiently rich dynamics.  ESN is
normally implemented with leaky-integrated, non-spiking, discrete-time and
continuous-value artificial neurons (see Fig. 2 the network structure). To
illustrate the technical details, here we use the notations by [33]. Consider
a temporal processing task, where the input signal is
$\mathbf{u}(n)\in\mathbb{R}^{N_{u}}$ and the desired target signal is
$\mathbf{y}^{target}(n)\in\mathbb{R}^{N_{y}}$, given $n=1,...,T$ with $T$
being the total number of discrete data points. The goal is to generate an
output signal $\mathbf{y}(n)\in\mathbb{R}^{N_{y}}$ that matches
$\mathbf{y}^{target}(n)$ as optimally as possible by minimizing the error
between the two signals (e.g., Mean-Square Error, MSE). The simplified update
equations of the reservoir part in ESNs are given by:

$\mathbf{\tilde{x}}(n)=tanh(\mathbf{W}^{in}\mathbf{u}(n)+\mathbf{W}\mathbf{x}(n-1)),$
(1) $\mathbf{x}(n)=(1-\alpha)\mathbf{x}(n-1)+\alpha\mathbf{\tilde{x}}(n),$ (2)

where  $\mathbf{\tilde{x}}(n)\in\mathbb{R}^{N_{x}}$ is the update at time step
$n$, $\mathbf{x}(n)\in\mathbb{R}^{N_{x}}$ is the state vector of the reservoir
neurons (also known as the resulting states or the echo of its input history
[15]), $\mathbf{W}^{in}\in\mathbb{R}^{N_{x}\times N_{u}}$ and
$\mathbf{W}\in\mathbb{R}^{N_{x}\times N_{x}}$ are the weight matrices of the
input-reservoir connections and the recurrent connections inside the
reservoir, respectively. $tanh()$ is the non-linear activation function
applied element-wise. $\alpha$ is the leaking rate that mainly controls the
speed of the dynamics.

The readout layer is normally linearly defined as:

$\mathbf{y}(n)=\mathbf{W}^{out}\mathbf{x}(n),$ (3)

where $\mathbf{y}(n)\in\mathbb{R}^{N_{y}}$ is the output vector and
$\mathbf{W}^{out}\in\mathbb{R}^{N_{y}\times N_{x}}$ is the weight matrix of
the reservoir-readout connections. Alternatively, one can also introduce a
bias value in both reservoir and readout, as well as integrate the input
signal directly to the readout layer. In this case, $\mathbf{u}(n)$ in Eq. 1
becomes $[1;\mathbf{u}(n)]$ and $\mathbf{x}(n)$ in Eq. 3 becomes
$[1;\mathbf{u}(n);\mathbf{x}(n)]$, where $[\cdot;\cdot]$ represents
concatenation. A brief training procedure is shown in Alg. 1.

Algorithm 1 Simplified procedure of ESN training.

1:Initialize the network by generating random $\mathbf{W}^{in}$, $\mathbf{W}$.
It is common to use uniform distributed randomization $U(-1,1)$.

2:Run the model with input signal $\mathbf{u}(n)$, $n=1,...T$, it will
generate the same length of the reservoir states $\mathbf{x}(n)$ by Eq. 1-2.

3:Collect all $\mathbf{x}(n)$, then calculate and get the output signal
$\mathbf{y}(n)$ by Eq. 3.

4:Minimize the MSE between $\mathbf{y}(n)$ and $\mathbf{y}^{target}(n)$ using
techniques such as linear regression. This should obtain a well-trained
$\mathbf{W}^{out}$.

5:Take unseen data $\mathbf{u}_{test}(n)$ and obtain the predicted and/or
generated output $\mathbf{y}_{test}(n)$.

During conventional ESNs training, $\mathbf{W}^{in}$ and $\mathbf{W}$ remain
unchanged, and only $\mathbf{W}^{out}$ is trained in order to minimize the
error between the network output and the target output (teacher signal),
usually by using linear regression such as ridge regression [34].
Alternatively, many new proposed learning rules for ESN training exist,
including but not limited to online FORCE learning [35], weights pre-training
[36], gradient-based training [37], and evolutionary learning [38]. For
details on training a RC model, please refer to section 3.

Echo state property. An essential condition (requisite) that a standard ESN
must meet is the echo state property (ESP), which ensures a condition of
asymptotic state convergence of the reservoir. This property is under the
influence of both the reservoir and the given input. On one hand, ESP is an
algebraic property that is controlled by the reservoir’s weight matrix
$\mathbf{W}$. It has been mathematically analyzed in [15, 39, 33] that the
spectral radius ($SR$, i.e., the maximum eigenvalue of $\mathbf{W}$) smaller
than unity ensures ESP in most situations. As a result, many RNN-based RCs in
literature consider $SR<1$ as a necessary condition to make the models work
(see the introduction section in [16]). However, it had been proved in [39]
that $SR<1$ is neither sufficient nor necessary for the ESP. The author claim
that  “it is not required to scale the spectral radius below 1, and there is
no general benefit in scaling the spectral radius toward the Edge of Chaos”.
The same paper also proposed new sufficient conditions for the ESP. Please
refer to [39] for mathematical details. On the other hand, ESP has been
empirically studied in the presence of driving inputs of varied strength [40],
without looking at the mathematics. [40] shows that for an input-driven
reservoir and a proper input scaling, the actual range of ESP validity (i.e.,
$SR$), is much wider than what is covered by the above literature conditions.

Memory capacity and edge of chaos. As a special type of RNNs, ESN also has the
characteristic of short-term memory. Analytical results that characterize the
dynamical short-term memory capacity of reservoirs were discussed in [41, 42].
Meanwhile, it can be found in a good deal of literature that reservoirs are
claimed to work best when they are tuned to operate at the so-called “edge of
chaos” [43, 44]. Here, the edge of chaos refers to a region of parameter
settings which makes the dynamical system operates at the boundary between the
chaotic and non-chaotic behavior. However, this is also a misnomer, as claimed
by Jaeger in [45, 39] that the “edge” in question here is the edge of the ESP,
not the edge of chaos. For a detailed discussion, please refer to [46].

###  2.3 Liquid State Machines

Figure 3: Simplified structure of Liquid State Machine (LSM). Note that LSM is
implemented by spiking integrate-and-fire neurons, rather than non-spiking
artificial neurons used in ESN.

Liquid State Machine (LSM) was proposed by [11] from the perspective of
computational neuroscience in order to study the brain mechanisms and model
the neural microcircuits. In contrast to ESN that uses non-spiking artificial
neurons, LSM is more biologically plausible as it is based on the Spiking
Neural Networks (SNNs) with recurrent reservoir structures. Inside the
reservoir, usually a 3D structured and locally connected network of spiking
integrate-and-fire (IF) neurons is randomly created and stimulated by external
input spike train signals (see Fig. 3). Intuitively, the reservoir in LSM is
often called liquid, since they follow a metaphor of excited states as ripples
on the surface of a pool of water [11].  Similar to ESNs in its form, here we
use $\mathbf{u}(t)$, $\mathbf{x}(t)$ and $\mathbf{y}(t)$ to represent the
input, reservoir state and output, respectively. The reservoir dynamic of LSM
is given by:

$\mathbf{x^{M}}(t)=\mathbf{L^{M}u}(t),$ (4)

where $t$ represents continuous time, $\mathbf{x^{M}}$ is the reservoir state,
$\mathbf{u}$ represents the input spikes, and $\mathbf{L^{M}}$ is the liquid
filter for input-reservoir state transformation. The readout is given by:

$\mathbf{y}(t)=\mathbf{f^{M}}(\mathbf{x^{M}}(t)),$ (5)

where $\mathbf{y}(t)$ is the output vector and $\mathbf{f^{M}}$ is a “memory-
less” readout map. The readout here can also be trained using simple
algorithms.

Separation and approximation properties. LSMs have two mathematical
preconditions, namely separation property (SP) and approximation property
(AP). These two properties ensure that the network has fading memory (i.e.,
echo state property in ESN). Specifically, SP addresses the degree of
separation between different internal states $\mathbf{x}$ caused by different
input $\mathbf{u}$ (condition is met if the liquid filter $\mathbf{L^{M}}$
satisfies the point-wise separation property), whereas AP addresses the
capability of the readout layer to produce the target outputs given different
liquid states $\mathbf{x}$ (condition is met if the readout map
$\mathbf{f^{M}}$ satisfies the approximation property). For the mathematical
basis of SP and AP, please refer to [11]. Overall, these two properties in
LSMs, together with the ESP and memory capacity in ESNs literature, ensure the
RNN-based reservoirs function properly.

###  2.4 Comparison of ESNs & LSMs

ESNs and LSMs are two similar RNNs with reservoir structures. The main
difference between them is that LSM used spiking IF neurons, while ESN is
based on non-spiking neurons. This makes LSMs more biologically plausible to
be used in investigating biological mechanisms of information processing in
the brain. In terms of model implementation, although both models show
noticeable advantages in reducing computation cost and training time, LSM,
with its biologically inspired characteristics and spiking implementation,
becomes more suitable for new types of hardware such as neuromorphic chips
[47]. Therefore, LSM is reported with several hardware designs and
applications.

ESNs show better flexibility in model modifications, as many variants of ESNs
were proposed to enhance the network performances (see Section 3). These
modifications are mainly to overcome the disadvantages of conventional ESNs.
The first drawback is that the fixed connection weights could limit the
performance of ESNs, since they are randomly initialized without the process
of tuning or optimizing. Moreover, it is shown that the improvement of ESNs
performance will reach a saturation as the size of the reservoir increases to
a certain amount. This means that only increasing reservoir size may not
result in a better performance. As a result, researchers have been focusing on
constructing multi-reservoir ESNs, such as multi-layered ESNs [48, 49, 50] and
parallel reservoir computing [51]. Meanwhile, several optimization approaches
have been proposed to fine-tune the networks weights and hyperparameters by
using either evolutionary algorithms [50, 52] or gradient based optimization
techniques [37]. Detailed discussions of recent ESN and LSM models are covered
in section 4.

##  3 Training a RC Model

Table 1: Various training techniques of RC.

Category of techniques |  | Training approaches |
---|---|---|---
Classic readout training |  |  | Linear regression
---
Ridge regression [34]
Online learning techniques |  |  | Least mean squares (LMS) [53, 54]
---
Recursive least squares (RLS) [55]
FORCE learning [35]
FORCE variations [56, 57, 58, 59, 60, 61, 55, 62]
Online gradient based training |  |  | LSM method [53, 54]
---
GD-based optimizer [37]
BackPropagation-DeCorrelation (BPDC) [15]
Back-propagation Through Time (BPTT)
ACTRNN [63]
Evolutionary learning techniques |  |  | Genetic algorithm (GA) [64]
---
Particle swarm optimization (PSO) [65, 52]
Competitive swarm optimizer (CSO) [50]
Evolino [66]
Biologically plausible learning techniques |  |  | Hebbian learning [67]
---
Spike Timing Dependent Plasticity (STDP) [68, 69, 70]
Intrinsic Plasticity (IP) [71, 72, 73, 74]

In this section, we will delve into an array of methodologies for training and
optimizing RC models (see Table 1). Beyond simply training the readout, we
will explore diverse techniques that aspire to enhance the construction of
reservoirs from various aspects, thereby enabling more effective and efficient
models.  These include but does not limit to (1) classical readout training
such as ridge regression [34]; (2) online learning such as least mean square
method and FORCE learning [35]; (3) pre-training such as particle swarm
optimization [52]; (4) online gradient based learning with back-propagation
[75, 76]; (5) evolutionary learning such as Evolino [66]; and (6) biologically
plausible learning techniques such as Hebbian learning [77].

###  3.1 Classical Readout Training

The original works of ESN and LSM state that the readout of a reservoir with
rich dynamics can be trained by using any statistical classification or
regression methods [16]. Using ESNs for example, it is recommended to apply
simple linear regression technique to single-layer readout. Again, we use the
notations by [33] for illustration. First, notice that Eq. 3 can be rewritten
and extended in a matrix form as:

$\mathbf{Y}=\mathbf{W}^{out}\mathbf{X}\approx\mathbf{Y}^{target},$ (6)

where $\mathbf{Y}\in\mathbb{R}^{N_{y}\times T}$ stands for the collection of
all $\mathbf{y}(n)$. Similar notation goes to $\mathbf{X}$ and
$\mathbf{Y}^{target}$. It is clear that $\mathbf{W}^{out}$ needs to be
optimized to minimize the difference between $\mathbf{Y}$ and
$\mathbf{Y}^{target}$. The most common technique is the ridge regression [34],
where $\mathbf{W}^{out}$ is obtained by:

$\mathbf{W}^{out}=\mathbf{Y}^{target}\mathbf{X}^{T}{(\mathbf{X}\mathbf{X}^{T}+\beta\mathbf{I})}^{-1},$
(7)

where $\mathbf{I}$ is the identity matrix with $\beta$ being the
regularization factor. Detailed implementation can be found in a practical
guide of ESN training in [33]. Once $\mathbf{X}$ is obtained in an off-line
way, one can tune $\beta$ to reach the best performance without any model
retraining. Ridge regression often shows sufficiency in many concrete tasks,
when the reservoir provides rich non-linear dynamics. Moreover, it is easy and
fast to train, which attracts many researchers coming from non-machine
learning backgrounds.

###  3.2 Online Learning Techniques

[33] provides many empirical solutions on how to produce a reservoir with good
initialization. However, problems still exist, as one cannot guarantee that
the reservoir is always well-initialized. Another effective way to solve this
problem is online adaptation. In this manner, a feedback loop between the
reservoir and readout is usually introduced.

####  3.2.1 Least Mean Squares (LMS) and Recursive Least Squares (RLS) Methods

Originated in the area of adaptive signal processing, LMS and RLS are the two
standard online learning methods for reservoir computing models [78]. The
mathematical description of LMS and RLS are presented in [53, 55]. To
illustrate, LMS is a gradient-based error minimization method in which an
error is exponentially discounted propagating back through time, yet this
method might be unstable because of the large eigenvalue spreads of the cross-
correlation matrix (i.e., $\mathbf{X}\mathbf{X}^{T}$). Moreover, it is
reported that LMS struggles to capture the history-dependent temporal data
[79]. Compared to LMS, RLS is more popular due to its robustness/insensitivity
to the effect of eigenvalue spreads mentioned above, as well as its faster
second-order convergence speed. Therefore, RLS method for RC has been widely
studied in [60, 59, 55]. Although RLS has advantages over LMS, it is more
computationally costly with $O(N^{2})$ time complexity, while LMS only
requires $O(N)$ in most situations [41], where $N$ is the number of variables.
Having said that, RLS is employed by FORCE learning, which creates a new
branch of reservoir computing with regard to cognitive science and brain
mechanisms.

Figure 4: FORCE learning overview. The red color represents that the weight
connections can be modified during training.

####  3.2.2 FORCE Learning

It has been shown that RNNs often experience spontaneous chaotic activity, and
algorithms such as BPTT [30] are usually not able to converge if the network
exhibits chaotic activity. ESN models address the chaotic activity by ensuring
echo state property (ESP), so that the models do not operate in a chaotic
manner. Instead of avoiding spontaneous activity like ESNs, Fisrt-Order
Reduced and Controlled Error (FORCE) learning, which is perhaps the most
popular online learning method of RC, reveals that the results are better when
the reservoirs exhibit chaotic behavior before training [35]. By modifying the
synaptic strengths of the reservoir (either internal or external), models
trained with FORCE learning show the effectiveness of suppressing autonomous
chaotic activity while turning it into a wide variety of desired output
patterns.  Since the original mathematics are quite complex, here we aim to
provide a simplified description of the online readout training for
completeness [80]. Please refer to [35] for details.

Consider the reservoir state as $\mathbf{r}(t)$ (i.e., $\mathbf{x}(n)$ in
ESNs), the network output $\mathbf{z}(t)$ (i.e., $\mathbf{y}(n)$ in ESNs) is
defined as:

$\mathbf{z}(t)=\mathbf{w}^{T}\mathbf{r}(t),$ (8)

where $\mathbf{w}$ is the weights connecting reservoir and readout. Note that
here the output dimension is restricted to one, while it can be easily
generalized to multidimensional. Training of $\mathbf{w}$ happens at every
time interval $\Delta t$. Before updating at time $t$, the error is denoted
by:

${e}_{-}(t)=\mathbf{w}^{T}(t-\Delta t)\mathbf{r}(t)-f(t),$ (9)

where $f(t)$ is the predefined target function (i.e., $\mathbf{y}^{target}(n)$
in ESNs). The FORCE algorithm uses a modified RLS method to update the weights
by:

$\mathbf{w}(t)=\mathbf{w}(t-\Delta t)-{e}_{-}(t)\mathbf{P}(t)\mathbf{r}(t),$
(10)

where $\mathbf{P}(t)$ is a square matrix that is updated at the same time as
the weights according to

$\mathbf{P}(t)=\mathbf{P}(t-\Delta t)-\frac{\mathbf{P}(t-\Delta
t)\mathbf{r}(t)\mathbf{r}^{T}(t)\mathbf{P}(t-\Delta
t)}{1+\mathbf{r}^{T}(t)\mathbf{P}(t-\Delta t)\mathbf{r}(t)},$ (11)

and is initialized as:

$\mathbf{P}(0)=\frac{\mathbf{I}}{\alpha},$ (12)

where $\mathbf{I}$ is the identity matrix with $\alpha$ as constant. After
training, the error becomes

${e}_{+}(t)={e}_{-}(t)(1-\mathbf{r}^{T}(t)\mathbf{P}(t)\mathbf{r}(t)).$ (13)

Finally, the training will end when it reaches

$\frac{{e}_{+}(t)}{{e}_{-}(t)}\approx 1.$ (14)

The above modified RLS method is applied to suppress the output errors and
frequently adapt the weight matrices in the reservoir or readout (see Fig. 4).
This makes FORCE learning disparate from other traditional iterative training
schemes—the errors in FORCE learning are always small during training, even at
the beginning, suggesting that the aim is not to reduce errors but rather to
keep the errors small. When training is done, the network will autonomously
generate the desired output. As the author claimed, FORCE learning helps to
construct machine learning based RNNs that “generate complex and controllable
patterns of activity either in the absence of or in response to input”. It
provides an interesting link between computational and biological
neuroscience. In short, FORCE learning can be seen as a useful tool for
optimizing RC, and simultaneously, it presents a potential model that could
help understand biological neural circuits [35].

Figure 5: Generation and stability of complex spatio-temporal motor patterns
by FORCE learning (modified from [58]). Left panel shows the ten trials to re-
generate the target signal (in gray line) after training. Right panel shows
the same trials under the addition of a perturbation.

####  3.2.3 FORCE Learning Variations and Implementations

One of the disadvantages of FORCE learning is that the trained network is too
complex to analyze the neuron activities. Also, for complex real-world
problems such as speech recognition, networks trained by FORCE require many
more units to match the performance of gradient based networks, as reported in
[56].

Therefore, some studies have been published aiming at improving FORCE
learning, ranging from neuroscience (e.g., spiking networks) to physical and
hardware implementations. An “extended” FORCE [57], for example, was proposed
for more general internal learning by using the desired output to generate
targets for every internal neuron in the network.  This so-called “target-
generating” network is then improved by [58], in which the reservoir network
can preserve time information and generate complex and high-dimensional
trajectories even under high levels of noise (see Fig. 5). To this end, the
FORCE variations are still infeasible to be implemented as a spiking network.
Later, [59] proposed a full-FORCE algorithm. Compared with FORCE learning, it
requires fewer neurons, achieves significantly better performance in noisy
environments, and can also be applied to SNNs (see implementation in [60]).
For hands-on implementation, the full-FORCE has been realized using a Python
spiking network framework called Nengo [81]. Other recent improvements of
FORCE learning include (1) Two-Step FORCE that converges faster than the
original work [61]; (2) Transfer-FORCE learning which takes the advantages of
both LMS and RLS methods for better learning performance [55]; (3) R-FORCE
that aims to model multidimensional sequences [62].  Very recently, [82] built
an object-oriented, open-source Python package that implements a TensorFlow /
Keras API for FORCE.

###  3.3 Online Gradient Based Training

Reservoir based on non-spiking artificial neurons (e.g., ESNs) can be trained
by using gradient descent (GD) approaches. The LMS method mentioned earlier is
one of the candidates which is first proposed in [54], yet using such approach
shows poor stability even when trying to stabilize the network by adding
noise. [37] proposed a more stable version of GD-based reservoir to optimize
four hyperparameters: the input scaling, spectral radius, leaking rate, and
regularization parameter. Besides, BackPropagation-DeCorrelation (BPDC)
algorithm is another powerful method for online training of single-layer
readouts with feedback connected back to reservoirs. This algorithm is robust
to the random initialization of the reservoir weights, and it is also capable
of tracking quickly changing signals. Detailed discussions of BPDC are
presented in a survey [15]. Meanwhile, the classical Back-propagation Through
Time (BPTT) approach for RNN training can also be applied to RC model. It is
worth noting that a network architecture called Adaptive Continuous Time
Recurrent Neural Network (ACTRNN) [63] shows some similarity to GD-based RC.
Please refer to Section 6 for details.

###  3.4 Evolutionary Learning Techniques

Evolutionary algorithm. One of the disadvantages of the traditional RC models
is that the performance is highly reliant on the random initialization of the
weights and hyperparameters. While the optimal hyperparameters can be found by
grid-search techniques, using such techniques to find the optimal weights’
initialization is nearly infeasible [33], given a concrete task. Therefore,
instead of applying online learning rules, another possible direction, taking
inspiration from above, is to train (or pre-train) the reservoirs using
evolutionary algorithms (EA). When EAs are applied to evolve any type of
neural networks (including reservoirs) they usually receive the name of
neuroevolution.

Various types of EA can be used to evolve a reservoir, including (1) genetic
algorithms (GA) [64]; (2) particle swarm optimization (PSO) [65, 52] and its
variants [50]; and (3) artificial bee colony [83]. For example, a GA was
applied to a double-reservoir ESN for parameter optimization, yet without
optimizing weights of input and the reservoirs [36]. Inspired by LSTM [9],
another EA for RC was proposed called Evolino [66]. Evolino constructs units
that are capable of preserving memory for long periods of time, in which the
weights of the reservoir are trained using evolutionary methods. A performance
comparison of several EAs for RC are presented in [84].

Particle swarm optimization (PSO), which is an efficient and widely used
technique for finding optimal regions on complex spaces, has also applied to
reservoir weight optimization. The first two attempts of using PSO technique
include using a binary PSO to find the optimal reservoir-readout connections
[38], as well as a supervised PSO algorithm by [65] for better initializing
the input weights of RC. However, only a subset of the weights was tuned in
the latter model, due to the high computational cost. [52] further developed
the PSO algorithm for RC, where a portion of fixed weights in an ESN is pre-
trained via PSO. The results show improvements on model generation as well as
a faster convergence time, yet the network architecture is rather simple, and
some hyperparameters should be selected empirically and carefully. The latest
version of PSO based RC is the competitive swarm optimizer (CSO) for fault
diagnosis problems [50], which is a hybrid evolutionary algorithm combining
both a variant of PSO and local search (LS).

###  3.5 Biologically Plausible Learning Techniques

Hebbian learning and Spike-timing-dependent plasticity. Reservoir computing
takes inspiration partially from biological systems. LSM-based reservoirs, for
example, are implemented using spiking neurons. This indicates that some
biologically plausible adaptation methods can be applied for reservoir
training. Inspired by synaptic plasticity in human brains, the first attempts
would be to use Hebbian and anti-Hebbian learning to try to decrease the
eigenvalue spread in ESNs but failed [67]. Later, it is reported in [41] that
the reservoir trained by Hebbian learning “makes neurons prefer inputs that
are easy to predict and weaken connections from those that carry more
information”. In terms of LSM, the spike-time-dependent plasticity (STDP),
which is based on Hebbian learning and is often integrated with SNNs, is
reported to improve the separation property (SP) in some real-world speech
data [68]. STDP was further developed to reduce memory storage load to make RC
more hardware-friendly [69, 70].

Intrinsic plasticity. Another biologically plausible way of adaptation is
based on Intrinsic Plasticity (IP), which is an unsupervised learning rule
used for adapting the intrinsic excitability of the reservoir neurons. Here,
intrinsic excitability refers to a phenomenon called long-term potentiation,
in which brief and high-frequent stimulation tends to produce an increased
ability to generate spikes [85]. Early research of the integration of RC and
IP mainly focuses on reservoir pre-training and global optimizations [71, 72,
73]. In 2019, an IP with a local search scheme was proposed to improve the
flexibility of the IP rule by allowing hyperparameters such as learning rate
to be different [86].  In 2022, [87] applied IP learning to sucessfully tune
the parameter in MEMS-based RC.  As a side note, experiments on intrinsic
plasticity have shown that the output distributions of real biological neurons
may have different forms in different brain regions among various species
[74].

##  4 Recent Approaches in RC

###  4.1 Overview

RC has witnessed a significant development in recent years. On one hand,
traditional RC models such as ESN and LSM have been improved by many new
proposed models. Some of these models are built on top of the original ones to
achieve better performances, while several new architectures of RC have been
proposed to solve problems with increasing difficulties. On the other hand,
recent studies has demonstrated that the idea of reservoir (i.e., a dynamical
system that can generate high-dimensional and non-linear responses) can be
implemented by using various materials, such as electronic devices, physical
systems, and biological realizations.

In this section, we aim to cover recent approaches in RC from several
perspectives up until 2023, including ESN-based RC, LSM-based RC, dynamical
systems, and physical RC. These approaches are highly interdisciplinary and
are usually tested in several benchmark tasks. To introduce some, benchmark
tasks for pattern classification includes spoken digit recognition [88],
waveform [89] and handwritten digit image recognition [90]. Besides, the non-
linear Autoregressive Moving Average (NARMA) time series [53] was widely used
in time series forecasting, while a channel equalization benchmark [54] was
introduced to evaluate the RC performance on adaptive filtering and control.
In addition, temporal XOR task [43] and memory capacity task [42] were also
commonly used in studies that focus on system approximation and short-term
memory.

###  4.2 Recent Trends of ESN-based RC

ESNs represent one of the foundational RC models. The simplicity of
implementation makes them an approachable entry point of RC. Therefore,
improving and extending ESNs is not only a key pursuit within the RC
community, but its impacts extend far beyond, proving particularly influential
for researchers outside the field, and thus allowing more interdisciplinary
research. In the following, we review recent trends of ESN-based RC models.

####  4.2.1 Multiple Reservoirs

Figure 6: Architecture of DeepESN. For the readout organization for DeepESN,
at each time step the reservoir states of all layers are used as input to the
output layer. Figure modified from [49].

Deep ESN. Apart from the performance saturation problems, there is another
limitation in the conventional ESN, i.e., the single large-scale reservoir is
poor in simultaneously dealing with different timescales [91]. In this
concern, some studies started to investigate multiple timescale dynamics of
reservoir structure, as it has been found that stacking recurrent networks
with different topologies can generate multiple timescales at different layers
[49].

In 2016, [49] proposed a deep reservoir computing model to achieve
hierarchical timescale representation. This model, called deep Echo State
Network (DeepESN), stacks multiple reservoirs one on top of the other, as
shown in Fig. 6. Mathematically, consider $N$ the number of reservoir layers.
The update equations, extended from Eq. 1-2, is given by:

$\mathbf{\tilde{x}}^{(i)}(n)=tanh(\mathbf{W}^{(i)}_{fwd}\mathbf{x}^{(i-1)}(n)+\mathbf{W}^{(i)}_{rec}\mathbf{x}^{(i)}(n-1)),$
(15)

$\mathbf{x}^{(i)}(n)=(1-\alpha^{(i)})\mathbf{x}^{(i)}(n-1)+\\\
\alpha^{(i)}\mathbf{\tilde{x}}^{(i)}(n),$ (16)

where  $\mathbf{\tilde{x}}^{(i)}(n)$, $\mathbf{x}^{(i)}(n)$ and $\alpha^{(i)}$
are the update, the reservoir state vector, and the leaking rate at layer $i$,
respectively. $\mathbf{W}^{(i)}_{fwd}$ is the weight matrix connecting layers
$i$-1 and $i$, and $\mathbf{W}^{(i)}_{rec}$ is the weight matrix of the
reservoir at layer $i$, $i=1...N$. The number of input layer is denoted by
$i=0$ and $\mathbf{x}^{(0)}(n)=\mathbf{u}(n)$. At each time step $n$, the
composition of the states in all the reservoir layers $\mathbf{x}(n)$ is given
by:

$\mathbf{x}(n)=\langle\mathbf{x}^{(1)}(n),...,\mathbf{x}^{(N)}(n)\rangle.$
(17)

Experiments show that the deep RC structure can achieve (1) multiple timescale
representation, where the timescales are ordered along the network’s
hierarchy; (2) multiple frequency representation, where progressively higher
layers focus on progressively lower frequencies. Additionally, when there is a
perturbation at the input, the effects of this perturbation last longer for
higher layers in the stack, and this differentiation is drastically attenuated
when input is provided to every layer. Therefore, having deeper layers while
increasing distance from the input is a key architectural factor for obtaining
a time-scales separation.

DeepESN shows potential for designing more efficient RC learning algorithms
used for sequential and temporal data processing. From 2022, A branch of deep
ESN is recently studied by the research team of Tanaka et al. modified the
deep network architectures were proposed, combining with other techniques such
as sequence resampling [92] in 2022 and Hodrick–Prescott filter [93] in 2023.
These model are claimed to have high prediction performance in time-series
prediction tasks with relatively low training cost.

Deep Fuzzy ESN. In 2019, [48] proposed a novel deep ESN model with fuzzy
tuning called Deep Fuzzy ESN (DFESN). Here, two reservoirs are stacked, where
the first reservoir is applied for feature extraction and dimensional
reduction, and the second one is used for feature reinforcement based on fuzzy
clustering. In other words, the output of the previous reservoir was extracted
as features for the next reservoir input, followed by a feature reinforcing
process performed by fuzzy clustering for classification enhancement. In
DFESN, back propagation is no longer necessary, since the feature
reinforcement process can be considered as a layer-wise fuzzy tuning that
replaces the back propagation algorithm with lower computational costs. As
claimed by the author, input samples are clustered more easily, thus improving
the final classification performance.

####  4.2.2 ESN with Evolutionary Algorithms

Multi-layered echo state network autoencoder. Autoencoder (AE) is a type of
common feed-forward network for dimensionality reduction and feature
detection, in which non-linear transformations are performed in each hidden
layer to regenerate a new effective data representation from the originals.
This technique was introduced to RC area by [94] as the first recurrent and
non-gradient descent-based AE in the literature. In [94] an autoencoder was
implemented by using multilayered ESN, with a bi-level evolutionary algorithms
for optimizing the network architecture and weights. Particularly, PSO was
applied for the bi-level optimization, where the first level is the
architecture determination and the second one is the weights optimization.
Classification results on various benchmarks showed that the performance of
the evolved model is improved compared with the conventional ESN as well as
other CNN or SVM based models.

Competitive swarm optimizer. Pre-training an ESN using PSO introduces some
extra hyperparameters, which are usually determined empirically. Furthermore,
when dealing with high-dimensional optimization tasks, PSO is likely to
experience stagnation or premature convergence [50]. To address this, [50]
designed a deep ESN model with a competitive swarm optimizer (CSO) and used it
for fault diagnosis—a precise classification task. CSO avoids the problem of
optimizing too many parameters at once in PSO with its powerful particle
update rule: the particles are updated by evaluating a pre-defined fitness
function, and the winner particle will go straight into the next iteration.
For the implementation, CSO is combined with a local search technique to
further optimize the deep ESN structure. The work shows that deep reservoir
networks based on evolutionary algorithms are suitable not only for time
series prediction but can also be used to deal with classification problems
with adequate results.

####  4.2.3 Other Types of ESNs

Non-linear functions readout. As mentioned earlier, single reservoir ESN may
not be able to create rich enough non-linear dynamics. [91] proposed a new
method called Non-linear ESN based on non-linear functions and successfully
decreased the internal states of the network while increasing dynamic
complexity, thus reducing the computational load.  Specifically, recall that
$\mathbf{x}(n)$ is the internal reservoir state vector ($\mathbf{x}$ for
short), while in this method, it is replaced by a non-linear function:

$\mathbf{x}_{NESN}=\mathbf{f}(\mathbf{x})=a_{0}+a_{1}\mathbf{x}+a_{2}\mathbf{x}^{2}+\cdots+a_{n}\mathbf{x}^{n}.$
(18)

where $a_{n}$ being the constant. With this modification, non-linear
complexity and the learning capability are increased, which results in a
higher accuracy in time series forecasting. Moreover, as this method remains
simple structure design, it does not require extensive training, parameter
tuning or complex optimization process.

Small-world topology. Small-world (SW) network was first proposed by [95]. For
a regular topological network, each node is usually connected to its
neighboring nodes. For the connection to other randomly chosen nodes (not
adjacent), we denote the connection probability as $p$, where $p=0$ remain
regular topology, $p=1$ remain random topology and $p\approx 0.1$ as the SW
topology (see Fig. 18).

To further investigate the echo state property and learning performance of
ESNs, [96] presented an SW based ESN (SW-ESN). In this study, the input and
output nodes are segregated, and the reservoir remains as an SW topology; that
is, neurons connected to the input are different from neurons connected to the
output. Experiments showed that the SW topology enables the input to flow to
the output nodes, and the cluster organizations of the topology guarantee a
larger range of echo state property, thus improving the robustness and
learning performance of the ESN.

###  4.3 Recent Trends of LSM-based RC

Spike-timing-dependent plasticity (STDP). STDP is a local unsupervised self-
organizing learning rule based on Hebbian learning [69]. The main idea of STDP
is that if the firing neuron $A$ tends to induce/inhibit spikes from another
neuron $B$, the synaptic connection $\mathbf{w}$ from $A$ to $B$ is likely to
be potentiated/depressed [47]. In other words, the synaptic connection
$\mathbf{w}$ from $A$ to $B$ is potentiated if a causal order (i.e., the
presynaptic neuron fires before the postsynaptic neuron) is observed, or
depressed otherwise. As a spiking neural network, LSM was shown able to be
trained by this adopted learning rule in an online learning manner, and
therefore significantly reducing memory storage load and computational cost,
as well as making LSM becomes more hardware-friendly for physical
implementations [69, 70]. Specially, a recent hands-on implementation of LSM
based on [97] is realized using NEST simulator [98].

Evolutionary algorithms. Similarly to the small world topology, the percentage
connectivity indicates the connection probability between neurons within the
liquid. Finding a proper percentage of connectivity is then an important
factor for improving the accuracy of LSM. Too high/low connectivity will harm
the performance, which also suggests that there is an optimal connectivity for
a given task. Particularly, [99] proposed an evolutionary algorithm to
optimize the number of neurons and percentage connectivity on a single liquid.
Meanwhile, [100] used a covariance matrix adaptation evolution strategy to
optimize three parameters, i.e., percentage connectivity, weight distribution
and membrane time constant in one liquid. However, [101] pointed out that the
above algorithms “only perform parameter optimization in a single liquid and
do not optimize the architectures of LSM,” and proposed a Neural Architecture
Search (NAS) based framework to optimize both architecture and parameters of
LSM model. Furthermore, the presented framework introduced a three-steps
search for LSM, where the first step is architecture optimization, the second
step is the search for the number of neurons and the final step is parameters
optimization. Experimental results show that the proposed model achieves
comparable accuracy on classification tasks of the three datasets (i.e.,
MNIST, Noisy MNIST and FSDD).

###  4.4 Dynamical Systems

The key essence of RC lies in its approach to use large, fixed random
networks, i.e., the reservoirs, exhibiting a rich set of dynamical behaviors.
These reservoirs provide high-dimensionality and memory in which input data
can be transformed and stored, making it easier to model complex temporal
patterns and perform machine learning tasks. On the other hand, dynamical
systems, characterized by their temporal evolution and behavior, offer crucial
insights into the working of these reservoirs, shaping how we understand and
optimize them. In the following subsections, we show how RC provides a
practical framework for studying dynamical systems, while the theories
underlying dynamical systems give a solid mathematical foundation to the
operation of RC.

Figure 7: RC with dynamical systems. (A) Single-node time-delayed feedback
reservoir [102]. (B) Cellular automaton [103]. (C) DNA oscillator [104].

####  4.4.1 Single-node Time-delayed Feedback Reservoir

Definition. Classical RC models process a low-dimensional temporal input
through a high-dimensional reservoir state space. This high-dimensional state
space is achieved by creating many randomly connected artificial (ESN) or
spiking (LSM) neurons as a reservoir, so as to receive input data coupled into
the reservoir with weight (synaptic) connections. It turns out that RC can be
implemented by using only a single hardware node. In 2011, the concept of
using a single dynamical node as a reservoir to generate a high-dimensional
space, was introduced in [102]. In contrast to network-based reservoirs
consisting of many neurons such as ESN and LSM, single node RC shows great
simplicity especially for physical implementation, Theoretically, the proposed
delay system refers to non-linear delayed feedback system, which is a type of
dynamical system described by delay differential equations given by [105]:

$\frac{dx(t)}{dx}=F(t,x(t),x(t-\tau)),$ (19)

where $t$ and $x$ are continuous time and state variables, respectively. $F$
is the function of this system with $\tau>0$ the delay factor. This system is
usually employed by using electronic circuits with a feedback loop (see Fig.
7A).  The workflow is as follows:

  * •

A low-dimensional input signal (e.g., one-dimensional temporal signal) is
first processed using a time-multiplexing masking function, and modulates the
state of the node.

  * •

The single node samples the pre-processed input states and holds them for a
delay period of $\tau$.

  * •

Meanwhile, $N$ virtual nodes are set that equally divide $\tau$ with the time
interval of $\theta=\tau/N$, forming a delay line.

  * •

When the signal reaches the end of the delay line, it is fed back into the
node, influencing the node’s future states.

  * •

The current state of the node and the states stored in the delay line are then
fed to the readout layer with trainable weighted connections.

Extensions. The single-node delayed feedback RC was experimentally
investigated on spoken digit recognition task and NARMA time series prediction
task [88, 42]. Some variations of delayed feedback structure were proposed,
such as (1) Different ensembles of delay-based RC with several delayed neurons
by [106]. and (2) two circular connected time-delayed based reservoirs with a
longer delay line by [107].

####  4.4.2 Cellular Automaton

Another type of dynamical system that can be used as an RC is the cellular
automaton (CA). CA is a collection of a cell-grid of specified shape that
evolves (interacts with its neighbors and changes its state) through a number
of discrete time steps [108]. CA’s new state is determined by a pre-defined
set of update rules and neighboring cells, resulting in a rich dynamic (see
Fig. 7B).

The concept of CA system was further extended to RC. A series of work on
building CA-based RC was made by [109, 103], where an evolution rule was
introduced to create a space-time volume in the automaton state space (i.e.,
the reservoir).  The proposed CA system was reported to be suitable for
combining other types of discrete dynamical systems such as Boolean logic and
symbolic processing [110].  Recent studies on CA-based RC also include
modifications and extensions of network architecture [111, 112]; as well as
improvements of evolution rules in CA [113]. More recently, [114] explored the
advantages of critical spacetime patterns generated by elementary cellular
automata (ECAs) in reservoir computing, specifically focusing on the
distractor’s length in time series data and proposing asynchronously tuned
ECAs (AT-ECAs) to generate universally critical spacetime patterns.

####  4.4.3 Coupled Oscillators

Definition. RC models have been successfully applied in research areas such as
machinery, chemistry, biology and physical systems (see section 5). Among
these implementations, many RCs are built using coupled oscillators. A general
representation of the dynamics of coupled oscillators is given by an ordinary
differential equation:

$\frac{dx_{i}(t)}{dt}=F(x_{i}(t))+G(x_{1}(t),...,x_{N}(t)),$ (20)

where $i=1,...,N$ is the index of a total of $N$ coupled oscillators,
$x_{i}(t)$ is the state of $i^{th}$ the oscillator at time $t$, $F$ and $G$
are an isolated function and a coupling function. The rich dynamics are
provided by each oscillator and the interactions between them [23].  In the
following, we aim to review studies that utilize coupled oscillators as a
building block of RC. Additionally, we will discuss oscillation mechanism of
brain in Section 6.

Mechanical oscillators. The first category of RCs using coupled oscillators is
based on mechanical oscillators. [115] built a network with anharmonic (i.e.,
non-linear) oscillators, where the components include masses coupled linear or
non-linear springs, making the system power-efficient to solve a bit-stream
computation task and a spoken words classification task.

DNA oscillators. In the field of molecular computing, a deoxyribonucleic
oscillator (DNA) reservoir was first proposed in [104]. This RC consists of
coupled deoxyribozyme based oscillators. Specifically, a microfluidic reaction
chamber was used to construct a reservoir, since different DNA species can
interact (see Fig. 7C).  Here, the microfluidic reaction chamber is a specific
kind of chamber used to carry out chemical or biological reactions under well-
controlled conditions.  Rich transient dynamics were then generated in the
reservoir, where the reservoir state consists of the time-varying
concentration of various species inside the chamber. A signal-tracking task
was then performed by using a reservoir with three DNA species that exhibits
oscillatory behavior. Recent developments of DNA oscillators include a random
chemical RC model [116], where the random chemical circuits (i.e., DNA strand
displacement) provide complex non-linear dynamics, making them suitable for RC
implementation. A novel RC using DNA oscillators was reported in [117] which
solves the problem of the lack of readout layer in the previous work [104],
and was then demonstrated for the handwritten digit recognition and a second-
order non-linear prediction.

Chemical reaction networks (CRNs). Related to the DNA oscillators, the
chemical reaction networks also show the capabilities to RC implementations.
One of the initial studies on CRN-based RC model was presented in a
presentation [118], from which the reservoir dynamic is given by a set of
ordinary differential equations (ODEs), while the readout layer is to learn
the Hamming distance between input bit-streams. In 2022, the author further
proposed a chemical RC for single stranded DNA (ssDNA) analysis [119].
Additionally, [120] used a modular framework to implement a RC model. The main
advantage of this work, compared with the previous DNA oscillators [104], is
that molecular computing allows changing the size of CRNs on-the-fly. Another
new chemical RC architecture was proposed by [121], where the reservoir was
implemented through electrochemical reactions. Also, it is reported that the
Polyoxometalate molecule (POM) in this chemical RC increases the diversity of
the response current and thus improves their abilities to predict periodic
signals. POM-based RC was further integrated with the so-called single-walled
carbon nanotubes as a random dense network [122]. Adequate results were
obtained in tasks including waveform reconstruction, non-linear autoregressive
modelling and memory capacity testing.

Other RCs with oscillators. It is reported that oscillatory behavior can be
restricted to the phase domain [123]. This makes it possible to apply phase
oscillators that exhibit rich dynamics to RC [124]. A RC using two coupled
relaxation oscillators built on $VO_{2}$ switches was reported [125], where
the oscillators show high order synchronization that allows simulating the XOR
operation. Besides, RC can also be implemented by using spin-torque nano-
oscillators in neuromorphic computing [126]. See Section 4.5 for detail of
spintronic RC.

Table 2: Types of physical implementations of RC: components and applications.

RC Type | Name | Components / Methods | Benchmark tasks / Applications
---|---|---|---
Electronic RC | Analog Circuits |  | 1\. Various electronic elements
---
with digital hardware.
2\. Spiking circuit implementations.
| Spoken digit recognition [127].
---
Memory capacity estimation [127].
Time-series prediction [128].
ECG signal processing [128].
Non-temporal non-linear task [129].
Efficient spiking implementation [130, 131].
FPGAs |  | 1\. FPGAs board with stochastic logic.
---
2\. Recurrent SNN on FPGA.
3\. Parallel neuromorphic hardwares.
| Channel equalization problems [132].
---
Image and isolated digit recognition [133, 134].
Short input and waveform-
patterns classification [135, 136].
Memristor |  | Neuromemristive components:
---
1\. Double crossbar arrays.
Memristive devices without neurons:
1\. Random memristor networks.
2\. Memristor with volatility.
3\. Memcapacitors.
4\. Atomic switch networks.
| Time-series prediction [137].
---
Waveform pattern generation/classification [138].
Associative memory task [138].
Image recognition [139].
Photonic RC |  | Spatially distributed
---
Optical nodes
| 1\. Semiconductor optical amplifiers
---
(SOAs) with digital masking.
2\. Photonic crystal platform.
3\. Nodes with free-space optics principles.
| Optical packet header identification [140].
---
Spoken digit classification [141].
Logical function prediction [142].
Waveform prediction [143].
Memory capacity estimation [144].
| Time-delayed
---
feedback loop
| 1\. Opto-electronic feedback loop.
---
2\. All-optical reservoir.
3\. Coherently driven passive cavity.
| Non-linear channel equalization [145].
---
Spoken digit recognition [145].
Spintronic RC | Spin-torque oscillators | Magnetic tunnel junction. |  | Short-term memory task [146].
---
Macro-magnetic simulation [147].
Spin wave |  | Thin Yttrium iron garnet film between
---
a magneto-electric coupling layer
and a conductive substrate.
Temporal XOR problems [148].
Magnetic skyrmions |  | 1\. Nanoscale magnetic vortex.
---
2\. Skyrmion fabrics.
3\. Magnetic skyrmion memristor.
| Image classification task [149].
---
Handwritten digit recognition [149].
| Dipole-Coupled
---
Nanomagnets
| 1\. Magnetic nonodots array.
---
2\. Magnetic random access memory
(MRAM) technology.
| NARMA10 task [150].
---
large-scale RC implementation [150].
Mechanical RC |  | Mass-spring-damper
---
systems
| 1\. Mass-spring-damper simulating
---
softbodied robots.
2\. Soft robotic arm (octopus).
3\. Pneumatically driven robotic arm.
| Active shape discrimination [151].
---
Learning to emulate timers-
delays and parity [152, 153].
Robot crawling [154].
Foraging learning task [155].
Sensors |  | State Weaving Environment Echo
---
Tracker (SWEET) sensing.
Ion concentration analysis [156].
| Tensegrity robots and
---
Central pattern generator
| 1\. Tensegrity-based structure with tensile
---
elements and compressive elements.
2\. Spiking-based reservoir acting as CPG.
Locomotion and sensing tasks [157].
Quantum RC |  | Quantum circuits /
---
computers
| 1\. Analog quantum dynamics under
---
a time-dependent Hamiltonian.
2\. Nuclear magnetic resonance (NMR).
3\. Quantum circuits with quantum gates.
| Quantum-chaotic system-
---
implementation [158, 159, 160].

###  4.5 Physical RC

Recall that the key feature of RC models is to transform sequential/temporal
inputs into a high-dimensional non-linear dynamical space (i.e., the
reservoir). If the reservoir provides rich enough dynamics, the desired output
can be read out by using simple learning methods such as linear regression
(see section 3 for details). Therefore, in principle, any kind of “non-linear,
high-dimensional dynamical systems which satisfies some conditions”, has the
potential to be a reservoir.

In particular, RC has become popular in a wide range of research fields
focusing on hardware design; that is, recent trend of RC implementation have
shifted to many domains of physical reservoir computing (PRC) such as optical
systems, neuromorphic devices, chemical reactions, quantum computing, to name
a few.

Several reviews have tried to organize this highly interdisciplinary topic of
PRC. A comprehensive overview of recent PRC implementations was reported in
[23], while [161] focuses on the recent advances in photonic RC. A book series
with detailed and special issues on designing PRC was published [45]. In the
following sections, we briefly outline the recently proposed PRC models for
completeness. Readers may refer to the articles above for a more systematic
and theoretical understanding of PRC.

####  4.5.1 Electronic RC

Figure 8: Electronic RC. (A) Hardware LSM architecture using FPGAs. BRAM
refers to block RAM, where $W$ and $W^{\prime}$ are the old and updated
synaptic weights, respectively [134]. (B) RC implemented by memristors with
local connections [137]. (C) Memristor-based reservoir computing using GAs for
training [138].

Analog circuits. Various electronic circuits are potential components for
hardware RC. Previously, we discuss the single-node time-delayed feedback RC
that was first proposed by [102, 127]. In fact, this type of RC can be formed
by electronic circuits with other digital hardware elements [162, 163].
Additionally, a single-node RC implemented by using spiking circuits was also
reported [130, 131]. The advantages of analog circuit based RC include (1)
less hardware requirements and (2) power efficiency in spiking
implementations. Therefore, RCs based on analog circuits were successfully
applied to tasks including (1) spoken digit recognition and memory capacity
estimation [127]; (2) time series prediction and ECG signal processing [128];
and (3) non-temporal non-linear tasks [129].

Field-programmable gate array (FPGA). FPGA board, as a hardware friendly
element, has been proven to be suitable for RC implementations. An early
attempt at the combination of RC and FPGA board was reported by [164]. In
2014, an FPGA board with stochastic logic was used to implement RC for non-
linear time series prediction task [165]. The offline learning in the work
above was further modified to an online learning scheme [132], where the units
in the reservoir exhibit sigmoid activation and learn with a gradient descent
algorithm. In terms of spiking implementation, the conventional LSM models
were successfully built on FPGAs in an early research [133] (see Fig. 8A), as
well as some models featured with parallel processing [134] and STDP learning
rule [166]. In fact, FPGA-based RC have shown (1) better re-configurability;
(2) much faster processing speed; (3) less energy costs compared to general
CPUs; and (4) more biologically plausible (SNN-based LSM). Several benchmark
tests were taken, indicating FPGAs based RC can be applied to (1) channel
equalization problems [132]; (2) image and isolated digit recognition [133,
134]; and (3) short input and waveform patterns classification [135, 136].

Memristive RC. A memristive device, or memristor, is a type of passive circuit
element that maintains a relationship between the time integrals of current
and voltage across a device. Some studies of RC focus on using memristive
elements. Here, the property of memristive elements is different from other
circuit ones, since they vary the resistance depending on the current flow at
different times. There are two main types of memristive components suitable
for RC. The first type is based on neuromemristive circuits. Specifically,
memristors are used to model the synaptic plasticity between neurons. [137]
successfully built an ESN based on memristor (see Fig. 8B), yet the
performance was worse than the conventional ESN in terms of time-series
prediction task. Other studies include using double crossbar arrays as
reservoir in ESNs [167] and LSMs [168].

Another branch of studies has shown that memristive devices without neurons
can also generate rich non-linear dynamics for RC implementations. The first
attempt was made by [138] for a wave pattern classification task, where a
memritstive topology was applied as a reservoir (Fig. 8C). In 2017, another RC
model using dynamic memristors for hard digits recognition was reported [139].
Other memristive networks were also explored to have potential of constructing
RC, such as (1) random memristor networks [169]; (2) memristor with volatility
[170]; (3) memcapacitors [171] and its hierarchy extension [172]; and (4)
atomic switch networks [173].

####  4.5.2 Photonic RC

Optical computing is another paradigm suitable for RC implementations, where
the complex non-linear and high-dimensional dynamics can be achieved in the
intensity and phase of the optical field. A wide range of studies aim to
uncover this specific area of RC. In principle, there are two main directions
of photonic RC implementations: (1) spatially distributed optical nodes, and
(2) time-delayed based photonic RC. For more detailed investigation and
discussion, please refer to two comprehensive reviews in [161, 23].

Spatially distributed optical nodes. It has been realized that the fixed and
randomly connected topologies in the conventional RC models (e.g., ESN and
LSM) can be implemented by spatially extended photonic networks using
spatially distributed optical nodes. Perhaps the first optical RC was proposed
and subsequently developed in [174, 140]. Here, an on-chip network of
semiconductor optical amplifiers (SOAs) was constructed to efficiently compute
the tanh function in the reservoir. Later, a digital masking approach was
proposed to overcome the short time delay and high operation rate in the
previous photonic RC [175]. From 2014 on, several techniques based on optical
nodes were reported, including (1) photonic crystal platform [143, 144]; and
(2) nodes with free-space optics principles [176]. In fact, optical nodes RC
are not only operating with lower power consumption, but more importantly,
they are extremely fast in computation. In terms of benchmark tasks and
applications, RC with optical nodes were simulated and applied to (1) optical
packet header identificationand spoken digit classification [140, 141]; (2)
logical function prediction [142]; (3) waveform prediction task [143]; and (4)
memory capacity task [144].

Time-delayed feedback RC. In section 4.4.1, we review RC with single-node
time-delayed feedback loop. After the first electronic time-delayed reservoir
proposed by [102], optical devices were quickly used to implement such systems
[177, 89]. In specific, these pioneering systems applied opto-electronic
feedback loops, where the optical parts with long fiber provide non-linearity
and time-delay while the electronic parts play the role of input processing
and output extraction [161]. Later, the electronic parts of the opto-
electronic based RC were replaced by active optical devices (i.e., SOAs or
fiber coupler) [178], forming an all-optical delayed based RC. Moreover, it is
reported that a significant improvement of high-speed, low-consumption can be
achieved by using passive devices (see Fig. 9D). An example is a RC with a
coherently driven passive cavity proposed in [145], in which a simple linear
fiber cavity was used as a reservoir to solve tasks such as non-linear channel
equalization and spoken digit recognition with remarkable performance.

Figure 9: RC implemented with electronic and optical elements [161]. (A) Opto-
electronic RC system. The optical (electronic) path is depicted in red (blue)
color. (B) Readout layer used in photonic RC by [178]. (C) All-optical RC
based on a semiconductor laser subject to delayed optical feedback. (D) All-
optical RC based on passive devices.

####  4.5.3 Spintronic RC

Spintronics, or spin electronics, is a branch of physics (particularly
condensed matter physics) and nanotechnology that uses the intrinsic spin of
the electron and its associated magnetic moment, in addition to its
fundamental electronic charge, in solid-state devices. It is reported
independently that several spintronic elements can be candidates for RC
implementations, including but not limited to spin-torque oscillators, spin
waves, magnetic skyrmion; and Dipole-coupled nano-magnets. Spintronic RC were
reviewed in [23], as well as in [146] specially for spin-torque oscillators.

Spin-torque oscillators. The first experiments of building a spintronic RC can
be found in [179, 180], where the spin-torque oscillators were used to provide
non-linearity. Here, the so-called magnetic tunnel junction is the key
component and is used as a reservoir. The advantage of spin-torque oscillators
is that the whole neural network can be emulated by the fast magnetization
dynamics generated by simple components. The performances of spin oscillators
based RC were evaluated with short-term memory estimation experiments, and
were quantified by macro-magnetic simulation [146, 147].

Spin waves RC. Another branch of spintronic RC is using spin waves as a
reservoir. The first RC based on spin waves was proposed by [181]. A bit
sequence input was used and a stripe magnetic domain structure is introduced
in a continuous magnetic garnet film where spin waves propagate, thus
generating spatially distributed rich dynamics. In 2021, spin waves RC was
numerically evaluated in [148], and was further shown to be low power
consuming [182]. For the applications, spin waves RC in the above studies
shows state-of-the-art performances in numerical experiments of temporal XOR
problems and memory capacity tasks, once the model is properly tuned.

Magnetic skyrmions. Magnetic skyrmions are small swirling topological defects
in the magnetization texture, in which the stabilization and dynamics depend
strongly on the topological properties of skyrmions [183]. In 2018, the first
prototype of RC based on skyrmion fabrics was proposed in [184, 185]. Here,
the skyrmion fabrics refer to the phases that interpolate between single
skyrmions, skyrmion crystals and magnetic domain walls. Owing to their random
phase structures, they are claimed to be suitable for RC implementation.
Another application on RC based on magnetic skyrmion is to implement physical
RC based on a single magnetic skyrmion memristor (MSM) for image
classification task (i.e., handwritten digit recognition) [149].

Dipole-coupled nanomagnets. A novel magnetic nanodots array was reported for a
new way of RC implementations [150]. The proposed system is a static magnetic
system, in which the dynamics are enriched by increasing the number of
nanomagnets. Specifically, the reservoir is formed by dipole-coupled
nanomagnets (nodes). Similarly to the conventional RNN-based RC, some nodes
were connected to the input, while all nodes were connected to the output and
were then read out by the magnetic random access memory (MRAM) technology. A
good aspect of nanomagnets based RC is that the magnetic interconnections
solve the wiring problem of hardware RNN implementations, showing a great
potential to build a large-scale RC system. The system performance was
evaluated in the NARMA10 task with adequate results.

####  4.5.4 Mechanical RC

Mass-spring-damper systems. Several mechanical RC models have been proposed,
including but not limited to soft robots and sensors networks. The idea of
employing robot’s body and its dynamics as a computational resource for RC, is
originated from the early works [186, 187]. The so-called mass-spring-damper
systems are used to replace the conventional neurons in the reservoir (e.g.,
artificial nodes in ESNs). Specifically, mass-spring-damper systems serve as
good models to simulate the biological bodies and soft-bodied robots, where
both systems show rich non-linear dynamics that can be used in physical RC
implementations. Inspired by the pioneering works above, [151] built a mass-
spring-damper system for active shape discrimination. From a more biologically
plausible perspective, [188] created a soft robotic arm based on an octopus.
The work implies that control can partially be outsourced to the physical body
and the interaction with the environment without being processed by the brain
or a controller. The authors further made a series of work on the octopus-
inspired robotic RC, showing that the implementations can learn to emulate
timers, delays and parity [152, 153]. For more applications of mechanical RC,
please see Section 5.

Sensors. RC had been shown potential for processing sensor data (see section
5). Yet some researchers like [156] argued that those reservoirs focusing on
sensing are often exploited in a somewhat passive manner, being a separated
post-processing component that receives data from sensors. Therefore, [156]
proposed the State Weaving Environment Echo Tracker (SWEET) sensing approach,
where the RC is considered the sensing element itself for novel sensing
applications such as ion concentration analysis.

Tensegrity robots and central pattern generator (CPG). For completeness,
tensegrity based robots and CPGs are briefly reviewed here.  To illustrate,
tensegrity refers to a stable structure that consists of tensile elements
connected by additional compressive elements [189], which is considered to be
as adaptive and resilient as the biological systems. Another concept, the
central pattern generator (CPG), is a neural network that can produce rhythmic
patterned outputs without relying on rhythmic sensory or central inputs [190,
191, 192, 193].  On top of the mass-spring systems, [157] first developed a RC
based on a tensegrity based structure and applied it to locomotion and sensing
tasks. Tensegrity structure can be further used as computational resources for
modelling biological structures like human bodies and cells due to its
stability. For example, CPG signals were generated by a tensegrity based RC
for locomotion [194, 195]. Another CPG-related RC was reported in [196], where
the FORCE learning algorithm was used to train a spiking-based reservoir that
acts as a CPG. [197] further showed that the biologically plausible tensegrity
robots are capable of adaptation to environmental changes.

####  4.5.5 Quantum RC

Quantum reservoir computing (QRC) is an intersection of quantum computing and
RC. Reviews on this topic can be found in [198, 199, 200]. The platform of QRC
was first proposed by [201]. The idea is to use analog quantum dynamics under
a time-dependent Hamiltonian, where the parameters are randomly chosen without
tuning.  Here, the Hamiltonian is a mathematical operator used to describe the
total energy of a quantum system. It plays a central role in the Schrödinger
equation, which describes how a quantum state evolves over time.  Several
improvements of QRC have been explored, including (1) boosting computing power
[202]; (2) enhancing memory capacity [203]; and (3) using Nuclear Magnetic
Resonance (NMR) [204]. Taking the advantage that any quantum-chaotic system
can be used for implementations, QRC are investigated in many other studies,
including (1) using quantum circuits with quantum gates [158]; (2) single non-
linear oscillator [159]; and (3) dynamical phase transitions [160].

###  4.6 Other RC models

RC with non-linear vector autoregression (NVAR). It turns out that RC can be
realized as a general, universal approximator of dynamical systems, in which
the RNN part contains non-linear activation neurons while the readout layer is
a weighted linear sum of the reservoir states. A novel concept was proposed by
[205] and [206], that RC with linear activation of neurons followed by a non-
linear readout is equivalent to a universal approximator. In this case, such a
RC would become mathematically identical to a non-linear vector autoregression
(NVAR) machine [207]. By identifying the limitations of random reservoir and
taking inspiration from [207], [208] proposed a so-called next generation
reservoir computing (NG-RC) model based on NVAR. The proposed model was built
without the requirements of random matrices and many meta-parameters, and the
feature vector of the NVAR was introduced equivalent to the readout of RC For
mathematical details, please refer to [208]. By applying NG-RC to three RC
benchmark tasks including Lorenz attractor prediction, the model showed faster
computational time while at the same time requiring only a small number of
sample and few meta-parameters for training. A possible application would be
using NG-RC to create a digital twin for dynamical systems.

##  5 Recent applications of reservoir computing

As a special type of recurrent neural network, RC avoids the main problem of a
difficult, unstable and resource-consuming training process. In the past
decade, however, various deep learning algorithms, took advantage of the
intricacies of gradient based RNN training with greater computational power
and finally became main-stream. This led reservoir computing research into a
niche for a few years. As scientists in various research fields have found
many new ways of RC implementations and applications (see Table 3), RC have
prompted renewed interest among researchers from disparate domains. This
section presents a comprehensive review of these recent trends, showcasing the
widespread applicability of RC from the realms of engineering and computer
science to the diverse fields of physical and social science.

Table 3: Recent applications of RC in various research fields.

Research Fields |  | Applications |
---|---|---|---
Biomedical |  |  | ECG, EMG and MCG signal processing [209, 210, 211, 212, 213, 214].
---
Medical images segmentation and classification [215].
Machinery |  |  | Active shape discrimination [151].
---
Temporal information processing [149].
Robotic crawling [154].
UAVs and sensors control [216, 217, 218, 219, 220, 221, 156].
Fault diagnosis [222, 223, 50, 48, 224].
Data Science |  |  | Series chunking and clustering [225, 226, 227].
---
Similarity learning [228].
Security |  |  | Attack detection [229, 230, 231, 232].
---
Specific emitter identification [233, 234].
Communications |  |  | Optical communications [235].
---
Network traffic flows analysis [236, 237].
Symbol detection in MIMO systems [238, 239].
Chemistry |  |  | Remaining useful lifetime prediction of PEMFC [240, 241, 242, 243].
---
Real-time gas concentration prediction-
using chemosensors [244].
DNA oscillators and Magnetic skyrmions [104, 120, 116, 117].
Chemical reaction networks [120, 121, 119].
Environmental |  |  | Wind forecasting [91, 245].
---
Wind power generation [230].
Audio and Speech |  |  | Speech recognition [88, 246, 247, 47, 248].
---
Music classification [249].
Finance |  |  | Stock market prediction [250].
---
Financial system modelling [251].

###  5.1 Biomedical

Electrocardiogram (ECG). A modified ESN is applied to cardiac monitoring
[209]. Specifically, the experimental data includes two classes of ECG signals
from MIT-BIH databases with highly imbalanced number of instances. The
reservoirs are used as patient-adaptable classifiers.  These classifiers can
not only produce accurate results, but also show the potential to implement
ECG classifiers by using neuromorphic hardware with spiking neural networks.
Similarly, [210] used an ESN for abnormal cardiac activity detection (Fig.
10A). The main objective is to apply an ECG monitoring model in Medical
Internet of Things (MIoT) devices with fast speed and low power consumption.
The proposed RC model shows better performance and generalization in AHA and
MIT-BIH-SVDM datasets than patient-adaptable methods, and also suggests that
RC can be implemented in wearable wireless devices. Besides, in the problem of
ECG signal denoising, a single-node RC is applied to solve this problem by
minimizing the EMG signal that impairs the ECG signal [211].

In 2021, [252] reported a new type of reservoir for arrhythmic heartbeats
classification. Inspired by the pioneering work on the combination of organic
electrochemical transistors (OECTs) and reservoir computing, they implemented
dendritic networks using OECTs for real-time classification. In detail, the
reservoir is created by using coupled dendritic fibers. Once these fibers are
excited through the electrolyte, they create a strong enough non-linear
dynamic of the input signals. The proposed networks show the potential use for
biofluid monitoring and biosignal analysis with high accuracy, indicating that
bio-compatible computational platforms can interact with body and biological
analysis.

Electromyography (EMG). Some studies on EMG also apply RC models. [253]
proposed a scalable and reconfigurable neuromemristive reservoirs architecture
for EEG and EMG signal analysis. A further EMG application using LSM model was
proposed by [212], where the EMG signal collected by the surface EMG (sEMG)
sensors are classified by an LSM-based neuromorphic hardware. In 2021, another
spiking RC model that applies CRITICAL plasticity rule [254] for synaptic
connection optimization was proposed for hand gesture recognition [255] (Fig.
10B). The author also proposed a novel approach to evaluate and convert the
raw EMG signals to spikes encoding.

Magnetocardiography (MCG). Checking the ECG is not possible for everyone.
Alternatively, magnetocardiography (MCG) signals can be detected by measuring
the magnetic field produced by the electrical currents in the heart and can be
converted into ECG signals. [213] built the first physical RC model for MCG
monitoring. Specifically, the noisy sensed MCG signals take as input to the
reservoir (i.e., the spintronic sensors, see Section 4.5) and the output is
the predicted ECG signals (Fig. 10C). This lightweight RC model is claimed to
be much power-saving and lower memory-required while achieving comparable
performance with a deep learning based filtering approach [214]. A similar
model was later proposed by [256] to extract ECG signals from MCG signals.

Other biomedical applications of RC. In early research of molecular reservoir
computing, the coupled deoxyribozyme oscillators is shown to be a type of
reservoir [104]. This refers to DNA reservoir computing (see Section 4). In
addition to DNA based RC, a medical image classification with distributed
representations on cellular automata RC was reported by [257] (see cellular
automata RC in Section 4). Besides, a recent study on spatio-temporal feature
learning used RC model for T-cell consistent segmentation [215]. Instead of
only applying a single reservoir, the model used multiple reservoirs for image
segmentation and classification, where each reservoir focuses on a specific
area of the image to obtain local interactions.

Figure 10:  (A) Ventricular heartbeat classifier using RC [210]. (B) Gesture
recognition using a plastic reservoir (EMG signal processing) by [255]. (C)
MCG Monitoring using physical RC with spintronic sensors acting as physical
reservoir nodes [213].

###  5.2 Machinery

Figure 11:  (A) Non-linear mass-spring-damper systems as a RC proposed by
[186] (middle panel). Pneumatically driven, modular robot arm as a RC proposed
by [258] (right panel). (B) A bio-inspired RC using octopus as soft robotic
arm by [188]. (C) A foraging learning task [155]. A vehicle robot was placed
in an environment with obstacles and was directed toward the goal. The system
used FORCE learning to generate a coherent signal output from a living
neuronal culture.

Robotics. In section 4, we gave a brief review of mechanical RC. The so-called
mass-spring-damper systems were built as the first type of mechanical RC by
[186, 187] (see Fig. 11A). This new type of RC was then used for active shape
discrimination [151]. Meanwhile, [188] made a series of works on the octopus-
inspired soft robotic RC (Fig. 11B), showing that the implementations can
learn to emulate timers, delays, and parity [188, 152, 153]. Another
application of robotic RC was reported to learn and reproduce various end
point trajectories by using a new RC-based soft robot system with a highly
complex pneumatically driven robotic arm [258].  RC has also been applied in
robotic crawling by using the origami – a traditional play of folding paper
into sophisticated and 3D shapes. [154] shows that an origami structure based
PRC can be designed to build a soft robotic controller for earthworm-like
peristaltic crawling.  [155] embedded FORCE learning into a robot to learn in
a living neuronal culture (i.e., foraging learning task), in which the robot
was placed on square fields with various obstacles and was directed toward the
target objects (see Fig. 11C).

Unmanned Aerial Vehicles (UAVs). RC can be applied to UAV systems.
Particularly in telecommunications, RC models were used in the so-called
cache-enabled UAVs for optimizing resource allocation over the LTE licensed
and unlicensed bands. The first attempt was to use ESN in such a system to
predict each user’s content request distribution and its mobility pattern when
limited information on the states of users and the network is available [216].
An LSM-based model was further proposed, which can predict more context
information of the users and thus improves the prediction accuracy [217].

Another branch of UAV applications includes one that uses a deep ESN based
reinforcement learning algorithm for UAV path planning by [218]. In this
system, each UAV uses an ESN to optimize paths and learns transmission power
at different locations. Besides, ESN can also be used to control rotorcraft
UAVs, which outperforms linear models in robustness to disturbances [259]. In
2021, [260] proposed a method for controlling flapping-wing UAVs in different
wind directions, where strain sensors are applied to measure the wind
movements, and a physical RC is used as a classifier to recognize the wind
stream from the sensor data (see Fig. 12A).

Sensors. RC models can also be integrated into wireless sensor networks
(WSNs). Generally, the sensor devices in WSNs are distributed and
computationally constrained, and the collected data usually consist of
temporal information, which makes RC inherently suitable for embedding on the
WSN devices [219]. One of the real-world WSN applications using RC is activity
recognition in Ambient Assisted Living (AAL) tasks [220]. Specifically, RC-
based multi-sensors were used for feature collection and extraction. The
sensed data were then further processed by an ESN which provides a good
activity recognition accuracy with low computational costs. In 2021, a bio-
inspired in-sensor RC was demonstrated to be effective for classifying short
sentences of language [221].

It is worth noting that although RC had been shown potential for processing
sensor data, some researchers like [156] argued that those reservoirs focusing
on sensing are often exploited in a somewhat passive manner, being a separated
post-processing component that receives data from sensors. Therefore, [156]
further proposed the State Weaving Environment Echo Tracker (SWEET) sensing
approach. Here, RC was considered as the sensing element itself for novel
sensing applications such as ion concentration analysis.

Fault Diagnosis. Fault diagnosis generally refers to the process of detecting
errors in physical systems while attempting to identify the source of the
problems. Built on the deep ESN architecture suggested by [49], [50] proposed
evolving deep ESN models for 3-D printer fault diagnosis, with a developed
version of particle swarm optimization (i.e., competitive swarm optimizer,
CSO). This RC model uses evolutionary optimization and is shown to be state-
of-the-art and computationally economic, which is a complement to deep
learning algorithms, rather than a competitor. Meanwhile, the same research
group proposed another solution for 3-D printer fault diagnosis [48, 224].
Specifically, deep ESNs were used to improve feature extraction performance,
where the features were reinforced throughout the hidden layers by using fuzzy
clustering as a tuning step. This low computational costing model also
provides the optimal solution in all experiments, with a total of 26 different
condition patterns in fault diagnosis data.

In addition, RC can also be applied to chemical fault diagnosis in the proton
exchange membrane fuel cell (PENFC) system. The first RC application in PEMFC
system diagnosis was made by [222], where a delayed feedback RC was used to
detect four fault types yet in the static operating conditions only. Instead
of processing voltage signal in the original data space, a newer variant based
on previous models was proposed, which performs abnormal detection in the
reservoir computing based model space (current-voltage model) without
requiring additional feature extraction [223].

Figure 12:  (A) A flapping-wing UAV for recognition of the wind direction in
which PRC approach was used to classify the wind stream from sensors [260].
(B) A dual-reservoir structure for chunking temporal information streams
[227]. (i) Chunking problems example. Three fixed chunks and noise (random
sequence between chunks) are repeated in the input sequence with equal
probabilities. (ii) The dual-reservoir structure. (iii) Selective readout
responses to the individual chunks, which are colored according to their
selectivity to the chunks.

###  5.3 Data Science

Image recognition. Image recognition stands as a prominent field within
computer vision and machine learning. While RC shows impressive performance in
finding and generating temporal features, it has also been adapted for image
recognition tasks, either alone or integrated with other techniques such as
deep networks with convolutional layers. Table 4 shows the performance
comparison between different RC models. To illustrate, when using RC alone to
deal with 2D or 3D inputs (i.e., images or videos), data are usually flattened
into 1D signals prior to feeding them into the reservoir, as discussed in
[33]. When combining with convolutional layers, these layers preprocess images
and videos, transforming them into intermediate representations which the
reservoir can process temporally [261, 262, 263, 264, 265, 266]. Although RC
models demonstrate competitive performances with other machine learning
methods like CNNs on simpler datasets such as MNIST, their performance
significantly declines when faced with datasets exhibiting higher spatial
complexity, such as CIFAR-10. This performance gap highlights the inherent
limitations of RC in dealing with complex spatial correlations, and
underscores the need for further exploration and investigations in this field.
On the other hand, studies have proved that convolutional neural networks are
likely to misclassify even if small perturbations are added to original
samples [17, 18, 19, 20, 21, 22]. Thus, this also highlights a strong
incentive for RC-based methods to tackle high-dimensional inputs with strong
2D/3D correlations, as it was shown that higher degrees of nonlinearity in the
model are related to more robust neural networks, and nonlinearity is where
the RC really shines.

Table 4: Comparison of the Accuracy (%) of recent RC models in image
recognition benchmarks.

Model Type | Model |  | MNIST | Fashion MNIST | CIFAR-10
---|---|---|---|---|---
Standard RC | [267] |  | 99.07% | - | -
Hybrid (CNN+RC) | [261] |  | 99.19% | - | -
[262] |  | 99.25% | - | -
[266] |  | 98.71% | 86.27% | -
[263] |  | 98.38% | 91.04% | 64.49%
Physical RC | [268] |  | 97.70% | - | -
[172] |  | 73.86% | - | 12.96%
[269] |  | 98.20% | 89.90% | -
[270] |  | 98.08% | - | -
Hybrid (CNN+PRC) | [271] |  | 98.90% | - | -
[272] |  | 99.03% | - | 60.57%

Clustering. One of the applications of RC in data science is clustering. As a
special case of clustering, time-series clustering introduces several
additional issues when compared with static data clustering. For example, the
lengths of time-series usually vary, and some of them may be infinite (e.g.,
video and audio sequences collected from CCTV cameras). Moreover, temporal
dependencies in different parts of a particular time-series cannot be captured
by making a fixed detecting window (i.e., the dynamical behaviors in sequences
contain to both short- and long-term correlation). As a result, similarity
measurement techniques, such as calculating Euclidean distance among temporal
data, are not inherently suitable for time-series clustering.

[225] proposed the first dynamic clustering algorithm using conventional ESNs.
The idea is to apply a clustering method inside every step of the reservoir’s
state update, where the author claimed that any unsupervised clustering
methods can be used in principle (e.g., k-means or any other iterative,
partitioning clustering methods). The proposed method overcomes the above-
mentioned issues and produces more compact clusters when applied to a hard
classification problem of detecting patients with eye disease in eye movements
datasets (saccades). In 2020, deep reservoir structure was introduced in time-
series clustering [226]. The proposed algorithm was applied to more common
benchmark datasets and showed better clustering quality than the previous
algorithm and static clustering methods.

Chunking. Related to clustering problems, some studies have been proposed for
sequence chunking [273, 274]. Here, the main difference between time-series
clustering and sequence chunking is that chunking finds the temporal
correlation between state variables, instead of clustering homogenous time-
series together based on a certain similarity measure (see Fig. 12B in the
upper-left panel). [227] used dual-reservoir networks that supervise each
other to mimic the partner’s responses to the given input. Here, a challenge
of chunking sequences with uniform transition probabilities, which can be
easily processed by humans in basal ganglia, was successfully solved by the
proposed model while conventional statistical approaches fail to chunk (see
Fig. 12B). This suggests that reservoirs can predict dynamical response
patterns to sequence input other than to directly learn transition patterns.

Similarity learning. [228] applied RC to learn the similarity between image
pairs with limited data. The reservoir here acts as a non-linear filter that
projects the images into a high-dimensional state space, in which the state
trajectories represent different dynamical patterns that reflect the
corresponding relationship of given image pairs. The proposed model was tested
on MNIST dataset and images taken from a moving camera. Compared to deep
Siamese Neural Networks, this RC model showed significantly better performance
in generalization tasks. The generalized combinations of relationships provide
robust and effective image pair classification.

###  5.4 Security

Figure 13:  (A) Diagram of Specific Emitter Identification (SEI) using RC
[234]. A multi-layer time-delayed feedback reservoir structure was introduced
to enable the linear classifier for emitter identification. (B) RC used for
symbol detection (modified from [275]).

Attack detection in Smart Grid. RC has been proven to efficiently solve false
data injection (FDI) and to improve the reliability in smart grid systems
[229]. The first attempt was to use a modified delayed feedback network (i.e.,
single-node time-delayed RC) as a reservoir combined with a multilayer
perceptron (MLP) as a readout for single-period attack detection [230]. This
RC model with MLP architecture produces a high attack detection rate (99%) and
shows strong robustness in various attack types. Later, the author extended
the pioneering work to the more challenging dynamic attack detection in smart
grid [231], where a bio-inspired learning rule called precise-spike-detection
(PSD) [276] is used for spiking reservoir training.

Regarding the attack detection, recent applications of RC include detecting
malware and micro-architectural attack, which is reported in [232] using a
CMOS-based RC neural network embedded in a 65nm CMOS chip.

Specific Emitter Identification (SEI). SEI is capable of extracting rich non-
linear characteristics of internal components within a transmitter to
distinguish one transmitter from another. Since the fingerprint of SEI cannot
be emulated, it is widely used in IoT devices to prevent MAC address spoofing
attacks. A reservoir with delay loops for SEI was first proposed by [233] and
further adapted to edge computing [234] where the RC architectures include a
digital loop (FPGA) and a photonic one (Fig. 13A).

###  5.5 Communications

Optical communications. In the high speed optical fiber communication systems,
RC was applied for digital equalization. [235] quantified the equalization
performance of the optoelectronics RC. Experiment results show that the
optoelectronics RC outperforms traditional equalizers under the same
transmission conditions, taking the advantage of its ring topology for better
correlation between adjacent data as well as its lower complexity and
computational cost.

Network traffic. [236] proposed a method for application identification for
network traffic by physical RC, which processes traffic flows as dynamical
time series data and enables fast and real-time identification. Another RC
application is reported by [237] for road traffic analysis.

Symbol detection in MIMO-OFDM systems. In wireless communication domains,
multiple-input multiple-output with orthogonal frequency division multiplexing
(MIMO-OFDM) is a key enabling technology in the 5G cellular network. Symbol
detection is an important technique due to the severe non-linear distortion
during transmission (Fig. 13B). Thus, an accurate estimation of MIMO-OFDM
channel is usually required. The first integration of RC and MIMO-OFDM systems
was proposed by [238]. Specifically, an ESN was used for system modelling and
predicting non-linear dynamics, where the MIMO-OFDM channel estimation is no
longer necessary. Further, inspired by deep RC architectures [49], [239]
extended the existing shallow RC to form a deep neural network [239, 275],
which significantly mitigates the frequency distortion.

###  5.6 Chemistry

Figure 14:  (A) An ESN was used to predict the remaining useful lifetime (RUL)
of Fuel Cells [243]. (B) A chemical RC implemented by a random chemical
circuit with different DNA species [116]. (C) An electrochemical-reaction-
based reservoir proposed by [121]. (i) Structure of the Polyoxometalate (POM)
molecule. (ii) Testing procedure.

Fuel Cells (FC). ESNs have shown their effectiveness on remaining useful
lifetime (RUL) prediction for proton exchange membrane fuel cell (PEMFC).
[240] developed the first RC model for FC prognostics using conventional ESN.
Later, several variants of RC model have been proposed for improving the
prediction performance. These include an ESN combined with ANOVA method [241],
as well as a multi-reservoir ESN [242]. However, recent research states that
the above proposals assume that FCs are operated in constant nominal operating
conditions [243]; that is, only the degradation is considered the factor of
the deviation of stack voltage. Another open problem is that the prognostic
results in long-term experiments show that prediction will become inaccurate
when disturbances occur.

Chemosensor. Metal oxide (MOX) based sensors are a common choice for tasks of
chemical detection, yet the time response of these chemical sensors is usually
excessively slow. It is stated that algorithms based on batch or sequential
measurements are not suitable for continuous sensing scenarios. [244] used RC
algorithms to overcome the slow temporal dynamics of the chemosensors and
applied RC for real-time gas concentration prediction by observing the
sensors’ time series in response to the changes in the composition of a gas
sample. Still, problems were reported, such as the drift of sensor response
over time.

Magnetic skyrmions. RC using magnetic skyrmions are reviewed previously in
Section 4, where the random phase structures of the skyrmion fabrics are
suitable for RC implementation [185]. Recent applications in this field is to
implement RC based on a single magnetic skyrmion memristor (MSM) for image
classification task (i.e., handwritten digit recognition) [149].

Coupled deoxyribozyme oscillators and DNA oscillators. Taking the inspiration
of DNA reservoir computing approach [104], [116] proposed a random chemical RC
model, where the random chemical circuits (i.e., DNA strand displacement)
provide complex non-linear dynamics, making them suitable for RC
implementation (Fig. 14B). The proposed model outperforms the previous
deoxyribozyme oscillator RC [120] in short and long-term memory tasks. Another
recent RC using DNA oscillators was reported in [117] which solves the problem
of the lack of readout layer [104], and then applied to a handwritten digit
recognition and a second-order non-linear prediction task.

Chemical Reaction Networks (CRNs). As mentioned earlier, reservoir’s dynamic
can be generated by a set of ordinary differential equations (ODEs). An
extension was proposed for single stranded DNA (ssDNA) analysis [119].
Besides, [120] used a modular framework for molecular computing to implement a
RC model. The main advantage of this work, compared with previous DNA
oscillators [104], is that molecular computing allows tuning the size of CRNs.
Another new chemical RC architecture was proposed by [121], where the
reservoir is implemented through electrochemical reactions since the chemical
dynamic is shown to be computing resources (Fig. 14C). As claimed by the
author, the Polyoxometalate molecule in the solution “increases the diversity
of the response current and thus improves their abilities to predict periodic
signals”.

###  5.7 Environmental

Wind forecasting and wind power generation. [91] proposed a RC based wind
speed and wind direction forecasting model. In fact, the proposed model is a
new type of non-linear echo state network, which is discussed in Section 4.
Instead of deterministic forecasting, a recent study focuses on probabilistic
wind power forecasting [245] by using a time warping invariant echo state
network [277]. In addition, the wind turbines were used as a major source of
power generation for smart grids, where the delayed feedback RC was applied
for attack detection [230].

###  5.8 Audio and Speech

Audio processing. Audio signals cover a wide range of temporal sequences such
as speech, sounds and music. In 2009, RC was first applied as a general
framework for non-linear audio processing by [278]. Three main potential
applications were proposed with simulations, including tube amplifier plugin
identification, non-linear audio prediction and music information retrieval
(MIR). RC was claimed suitable for non-linear audio processing because of the
inherently temporal processing capability. In terms of real-time audio
processing, a cascaded discrete-time RC was proposed for black-box system
identification [279]. Albeit much effort was made to reduce computation
consumption, the cascaded structure is stated considerably more complex to
tune than the conventional RC. In 2019, [249] proposed randomly weighted CNNs
for music classification. The proposed model shares similarity to RC, where
weight connections remain untrained during training.

Speech recognition. The earliest RC application of speech recognition was
presented in [88]. Here, an LSM-based RC with spiking integrate-and-fire
neurons was implemented recognizing isolated digits, where the readout is
trained using ridge regression. The problem of the model is that it requires
intermediate data storage for offline learning. Similarly to the work above,
[246] trained output neurons using back-propagation based MLPs. Inspired by
Hebbian learning, [47] further proposed a variant of Hebbian online learning
rule to train an LSM without requiring data storage for speech recognition. In
detail, the analog input speech signal is pre-processed by the Lyon passive
ear model and further converted into spikes by BSA algorithm [280] before
feeding into LSM. Other types of RC can also be applied to speech recognition.
For example, an ESN combined with extreme kernel machines was used for Arabic
speech recognition [247]. In 2020, a RC based on nano-oscillators was also
applied to TI-46 database [248].

###  5.9 Finance

Stock market prediction. A successful prediction of a stock’s future price
could yield significant profit. An early attempt of short-term stock price
prediction was reported in [281], who used an ESN as a basic network with the
Hurst exponent to select a persistent subseries with the greatest
predictability for training from the original training set. Instead of using
basic ESN, three RC network structures were investigated in stock price
prediction [250], including the small-world topology discussed in an earlier
section.

Financial System modelling. [251] used RC for financial system modelling. In
this study, an ESN was first applied to predict a pre-defined financial system
behavior. The model was further proved to effectively re-generate only the
required data based on limited known information.

##  6 RC with Brain Mechanisms and Cognitive Science

###  6.1 Reservoir in the Cerebral Cortex

RNNs have been shown to have rich, complex, non-linear and high-dimensional
dynamics. In the cerebral cortex, especially the prefrontal cortex (PFC),
massive recurrent connections of neurons were found, and it is progressively
recognized that some parts of the brain operate as reservoirs [282]. Moreover,
the cortex is shown able to extract the desired outputs (readout) from the
high-dimensional neural representations (reservoir). In this section, we
review studies on using RC to model the cerebral cortex.

####  6.1.1 Dominey’s Decade-long Research: the Birth of RC

Dominey et al. developed the first RC prototype in a series of neurocognitive
studies on corticostriatal systems [13, 28]. During the period of 80s-90s,
many researchers were focusing on the characterization of the fast eye
movements (i.e., the oculomotor saccade) in the corticostriatal system, which
refers to the interactions between cortex and basal ganglia [25].
Particularly, [26] examined the function of the corticostriatal system by
carrying saccade experiments on macaque monkeys. The experiments showed that
some neurons (1) have a preferred spatial saccade amplitude and direction; (2)
selective to response to a particular sequential order. In 2013, [24]
characterized this finding as mixed selectivity, which became one of the
important principles in RC and cognitive science.

Suggested by the two experiments of the corticostriatal saccade system [26,
27], the first corticostriatal RC model was built based on (1) a recurrent
prefrontal cortex (PFC) system (i.e., the reservoir), and (2) the reward-
related learning in PFC-to-caudate connections (i.e., the readout). Fig. 15
shows the architecture of the model. Since they found that the modification of
the recurrent connections are considerably computational costing, they decided
to initialize PFC layer (the reservoir) with a mixture of fixed inhibitory and
excitatory recurrent connections. The reservoir layer was then connected to
the caudate or striatum to obtain the readout. This pioneering RC model, as
the author claimed, can be seen as a dedicated temporal recurrent network
(TRN), which shows the inherent capabilities and sensitivity to temporal and
sequential structure by providing a rich spatio-temporal dynamic [283].

Figure 15:  The first RC model built based on a recurrent PFC system as
reservoir (modified from [28]).

Recent extension of TRN. Dominey et al. further proposed a series of works on
the previous corticostriatal RC model. These include a combination of RC and
neuro-physiological models of language processing [284, 285], as well as a
performance improvement of the RC learning algorithm [286]. Reader may refer
to a more detailed review of the corticostriatal RC model and its historical
developments in a review paper by [287]. In 2013, one of the important
cortical activities was obtained in randomly connected recurrent networks
(e.g., reservoir) and was then characterized as mixed selectivity [24] (see
the following section). In 2016, the representational power and dynamical
properties of mixed selectivity were investigated by training a RC model to
perform a complex cognitive explore-exploit task initially developed for
monkeys [288]. By comparing the neural activity of the reservoir and the
primate dACC neurons, it is found that not only mixed selectivity was observed
in the two types of neurons, but more strikingly, the distributions of neurons
were quite similar in terms of the epoch (explore/exploit), the task phase,
and the target choice, which strongly supports the argument that the cortex
behaves computationally as a reservoir (see Figures 3 and 4 in [288]).

####  6.1.2 Recent Studies of the Cortex

Dimensionality implies selectivity. One of the complex neural activity
phenomenons in PFC and in the model by [13] is that the firing rates of some
neuron populations were modulated by the combinations of conditions such as
spatial location and sequential order [289]. This cortical activity was then
characterized as mixed selectivity by [24] in an object sequence memory task.
Specifically, pure selectivity refers to neurons whose responses are selective
only to an individual task-relevant aspect, whereas mixed selectivity refers
to neurons whose responses are explained by a non-linear superposition of
responses to the individual task-relevant aspects. In the object sequence
memory task, monkeys were required to watch a sequence of two subsequently
displayed images. After that, they had to (1) recognize the two images under
distraction (recognition task), or (2) recall the order of the two images
(recall task). The dimensionality of the neural spaces was then estimated
(i.e., the minimal number of coordinate axes needed to specify the position of
all points in neurons’ firing rate space). It is observed that the
dimensionality was higher if neurons having mixed selectivity were included.
More importantly, the neural population was estimated to have a higher
dimensionality when the monkeys performed correctly on a trial. That is, in
the error trials, a collapse in dimensionality was observed, which impairs the
ability of downstream readout neurons to produce the correct response.
Moreover, [24] and [290] showed that RC models can be compared with a randomly
connected recurrent structure in the monkey prefrontal cortex, which can
generate high-dimensional mixed selective dynamics to assure the separability
in the downstream readout units. The higher the dimensionality of the
population coding, the better the performance on the task [289]. Regarding the
readout, it was reported that the brain implements mixed selectivity even when
it does not enable behaviorally useful linear decoding (i.e., simple linear
readout), suggesting that mixed selectivity may be the key of population
encoding for reliable and efficient neural representations [291].

Recent studies have shown that mixed selectivity not only plays an important
role in PFC, but in other parts of the brain. [292] found strong mixed
selectivity in the subiculum (i.e., the area between the entorhinal cortex and
the CA1 subfield), where individual neurons respond conjunctively to task-
related aspects including position, head direction, and speed. In 2022, mixed
selectivity was observed in the thalamus of a weakly electric fish [293].
Here, the mixed selectivity strategy was implemented to encode interactions in
the recurrent networks in pallium, which is related to courtship and rivalry
in terms of dominance in male-male competition and female-mate selection.

###  6.2 Neuronal Oscillations

Neuronal oscillations refer to the temporally structured activity generated in
mammalian brains, where neurons undergo periodic changes in excitability.
These oscillations had been found in neuron assemblies, a concept to describe
the behaviors by a population of neurons. In this section, we first present
the cognitive science research on neuronal oscillations, and then we discuss
the relationship between oscillations and RC models, followed by several
examples of the existing research. Contents are partially from [2].

####  6.2.1 Neuron Assemblies

Feed-forward circuits. It is widely believed that there are two frameworks of
processing in natural systems in the cortex [2]: (1) convergent feed-forward
circuits and (2) neuronal assemblies. In the framework of feed-forward
circuits, specific neurons fire to particular features, and the information is
propagated from the former layer to the next higher layer. In this way,
higher-level features (e.g., cognitive objects) are extracted through a multi-
layered structure. The encoding scheme here refers to spatial encoding, which
is well-suited for simultaneously presenting features such as images. However,
due to the lack of short-term memory functions, feed-forward circuits are less
apt to tackle the relations among temporally segregated events.

Neuron assemblies. On the other hand, a complementary framework in cognitive
brains is the neuronal assemblies. Unlike the feed-forward networks which
include explicit layered structures, the assemblies of neurons usually form
coupled recurrent networks with non-linear, high-dimensional and self-
organizing dynamic [2]. Relations among cognitive objects are translated into
the weighted connections between neurons; in other words, high-level features
are represented by the amplified reverberations (echoing) of neuronal
assemblies. With the reverberating responses, the rich dynamics provided by
the coupled recurrent connections have short-term memory (fading memory), and
become efficient to handle temporally related sequential events.

####  6.2.2 Blinding Problem

Problem of neuronal assemblies. One of the challenging problems of the neuron
assemblies is the blinding problem, which refers to the segregation of
simultaneously active assemblies. According to [77], if assemblies were solely
distinguished by enhanced activity (i.e., discharge rate) of the constituting
neurons, it becomes difficult to distinguish which of the more active neurons
actually belong to which assembly. Moreover, if the given objects share some
common features and overlap in space (e.g., blind source separation and
cocktail party problem), the corresponding feature-selective nodes would have
to be shared by several assemblies [2].

Solutions to blinding problem. A possible solution is multiplexing, in which
various active assemblies are segregated in time. Because of the discharge
rate of cortical neurons is relatively low (i.e., the integration needs time),
multiplexing becomes problematic if only discharge rate is considered for
distinguishing assemblies [294]. Therefore, it is only capable in a slow
timescale [295].

In the 1990s, Gray and Singer proposed that “neurons temporarily bound into
assemblies are distinguished not only by an increase of their discharge rate,
but also by the precise synchronization of their action potentials” [296, 297,
298]. They also predicted that neurons that respond to the same sensory object
might fire in temporal synchrony, with a precision in the millisecond range.
Synchronization by oscillation is briefly introduced in the following section.

####  6.2.3 Synchronization by Oscillation

Figure 16:  The spread of synchrony. Here, the blue neuron populations are
already synchronized, and they might entrain other neurons (those in green) to
become part of the same overall assembly after some point in time (see the
dashed red line), thus resulting from the spread of synchronized activity
through lateral connections. Figure modified from [299].  Figure 17:  The
Arnold tongue regime in coupled oscillators. This is a graphical
representation modified from [300] to illustrate the synchronization behavior
relating the difference in preferred frequency with increasing coupling
strength, which results in a ‘tongue’-shape of possible synchronization
regimes.

Definition of synchronization. The periodic changes of excitability of the
neurons are considered neuronal oscillations among different brain areas [2].
It has been identified that these oscillations vary in terms of the frequency,
ranging from approximately 0.05 Hz to 500 Hz [301]. According to [302],
synchronization by neuronal oscillations has been widely detected across
various natural systems, in which different oscillations can “coexist and
often synchronized to each other or nested into each other”. This observation
is called the spread of synchrony. As an example shown in Fig. 16, the
synchronized larger-scale populations can entrain other smaller local
assemblies with different oscillations to be overall synchronized. This spread
of synchronized activity was then believed to be a reinterpretation of the
represented objects [299].

Arnold tongue regime. One of the interesting observations of synchronization
behavior in coupled oscillators is the Arnold tongue regime. An early
experiment by Van Huygens revealed that the beats of pendulum clocks can be
synchronized when having the same timber; that is, if the preferred
frequencies of the oscillators are similar, weak mutual interactions are
enough for oscillatory synchronization [300]. This observation was then
summarized as Arnold tongue regime by [303]. As shown in Fig. 17, the coupling
strength should increase in order to assure a stable synchronization when the
preferred frequencies between coupled oscillators become increasingly
different, thus resulting in a tongue-shaped pattern. Synchronization would
become unstable if the difference between preferred frequencies exceeds a
critical point [300].

####  6.2.4 How RC Relates to Oscillations?

Recall that in most of the existing literature, RC models should normally meet
several requirements to be efficient and functional for various tasks. These
are also considered the properties of a RC model [23], which include (1) High-
dimensionality: low-dimensional inputs are mapped into a high-dimensional
space, which allows originally inseparable or temporal inputs to be linearly
separable as shown by the Cover Theorem [304]. (2) Non-linearity: non-linear
mapping transforms the input into linearly separable reservoir states which
can be read out by readout layer. (3) Separation property: a RC model should
be capable of separating different inputs into different classes, under small
fluctuations or in noisy environments [11]. (4) Fading memory: also known as
short-term memory or echo state property. This algebraic property eliminates
the effect of initial network condition. In other words, it ensures that the
reservoir state is dependent on recent-past inputs (reverberating responses),
but not distant-past inputs (responses faded).

In a RC review, [2] made a proposal to link the concept of RC in machine
learning and that of neuronal behaviors in the cognitive brain. Readers may
refer to the article for details. Based on the proposal, here we aim to
discuss how the dynamics of coupled oscillators in mammalian brains could be
exploited to accomplish the abovementioned four characteristics in RC.

High-dimensionality and non-linearity. The first fact is that the cortex is
reported to have consistent and random high-dimensional oscillations, which
refers to the “resting activity” [305]. Meanwhile, it is believed that brains
are likely to have an internal model of the external world (i.e., prior
knowledge, which can be updated by learning).  When the input comes in, the
input stimuli “activate” some feature-sensitive neurons, thus making the
dynamics of the network collapse into a stimuli-specific substate (e.g.,
oscillatory synchronization).  All of these may suggest that once a reservoir
enters a substate (i.e., synchronization), it is likely that the dynamics can
be tuned selectively to specific stimuli or generating specific output
signals.

Figure 18:  Neuroscientists suggest that neural oscillations reveal a highly
complex interplay of neural populations and local integrations by coupled
oscillations and synchronization, in which multiple timescales in hierarchical
processing streams can be achieved. (A) and (B) show examples of how these
mechanisms work on a computing and processing level. (A) GACTRNNs [63], a type
of neural network where neurons in a population can learn to simultaneously
represent temporally different primitives, by changing their timescales. (B)
An ESN with a small-world topology. The network rewiring probability is around
10% [96].

Separation property. As mentioned above, the stimuli-specific substate with
rhythmic oscillations, according to [2], “would have a lower dimensionality
and to exhibit less variance than the resting activity, to possess a specific
correlation structure and be metastable due to reverberation (rhythmic
oscillations) among nodes supporting the respective substate”. Note that all
these activities, including the stimuli-specific substates, are happening in a
high-dimensional state space; thus, different inputs can be well-separated and
classified even linearly, similar to the readout of RC models.

Fading memory. Moreover, fading memory refers to the short-term memory in RC
literature, in which the reservoir state should depend on recent inputs but
not distant-past inputs. From an oscillation point of view, there exist
several experiments in the cat’s visual cortex, which can support the