Control of synaptic plasticity in neural networks
The brain is a nonlinear and highly Recurrent Neural Network (RNN). This RNN
is surprisingly plastic and supports our astonishing ability to learn and
execute complex tasks. However, learning is incredibly complicated due to the
brain's nonlinear nature and the obscurity of mechanisms for determining the
contribution of each synapse to the output error. This issue is known as the
Credit Assignment Problem (CAP) and is a fundamental challenge in neuroscience
and Artificial Intelligence (AI). Nevertheless, in the current understanding of
cognitive neuroscience, it is widely accepted that a feedback loop systems play
an essential role in synaptic plasticity. With this as inspiration, we propose
a computational model by combining Neural Networks (NN) and nonlinear optimal
control theory. The proposed framework involves a new NN-based actor-critic
method which is used to simulate the error feedback loop systems and
projections on the NN's synaptic plasticity so as to ensure that the output
error is minimized.
Mohammad Modiri
# Control of synaptic plasticity in neural networks +

###### Abstract

The brain is a nonlinear and highly Recurrent Neural Network (RNN). This RNN is surprisingly plastic and supports our astonishing ability to learn and execute complex tasks. However, learning is incredibly complicated due to the brain's nonlinear nature and the obscurity of mechanisms for determining the contribution of each synapse to the output error. This issue is known as the Credit Assignment Problem (CAP) and is a fundamental challenge in neuroscience and Artificial Intelligence (AI). Nevertheless, in the current understanding of cognitive neuroscience, it is widely accepted that a feedback loop systems play an essential role in synaptic plasticity. With this as inspiration, we propose a computational model by combining Neural Networks (NN) and nonlinear optimal control theory. The proposed framework involves a new NN-based actor-critic method which is used to simulate the error feedback loop systems and projections on the NN's synaptic plasticity so as to ensure that the output error is minimized.

_Keywords:_ learning rule, neural network, nonlinear control, reservoir computing.
Proposed framework

In this section, we propose a novel brain-inspired learning rule. The main idea is to use a RL-based method to control synaptic plasticity. Fig. 1 illustrates the main ideas underlying this computational model that applied in a time series classification task.

In the proposed framework, synaptic plasticity reformulated as an optimal tracking problem. For this purpose, the one-hot output vector representing the true class of each time series pattern is imitated, similar to the length of each time series pattern, and is considered the reference trajectory as illustrated in Fig. 1 (C) bottom subplot. Since the reservoir is a special class of high-dimensional nonlinear dynamic systems, where both state and time are continuous. Therefore, continuous-time formulation of the HJB equation and ADP can be applied to derive the learning rule (optimal control law). The derived learning rule forces the proposed framework's output (predicted class in classification tasks) to mimic the reference trajectory (true class in classification tasks) that implicitly minimized the cross-entropy. The following subsections elaborate on the network architecture, and the learning rule in further detail.

### Network architecture and dynamics

The proposed framework consists of a feedforward and feedback pathway. The main parts of the feedforward module are:

* Input data encoding layer.
* Recurrent neural network (RNN).
* Linear classifier (output function).

The feedback module is composed of:

* Critic RC.
* Actor RC.

Fig. 1 shows the block diagram of the proposed framework architecture. The functionality of the proposed architecture is based on the following steps:

* The encoding layer projects input patterns \(x(t)\) into the RNN. In spiking neural networks, each real-value of a data vector is converted into discrete spike trains,
suitable for processing in the spike network as spike neural networks can only process discrete spike trains.
* In the pertaining process, a supervised learning algorithm like Gradient descent, Ridge, etc., is employed to adjust the decoder parameters associated with each training sample. Once the decoder is trained, it is considered constant during the reservoir training process.
* In the RNN training process. An RL-based algorithm proposed in subsection 1.2 is applied to make the RNN learn temporal relations between the input patterns \(\mathbf{x(t)}\) and desired output \(\mathbf{y(t)}\) by adjusting the parameters in the RNN using a novel ADP approach. In the proposed ADP mechanism, two RC-based actor and critic are utilized to approximate true reservoir parameters.
* Model recall on new data.

The feedforward pathway resembles the structure of traditional neural networks, however the methodology of the feedback pathway especially proposed in this study differs significantly from classical artificial intelligence approaches.

**Remark 1** For the sake of brevity, time dependence is suppressed while denoting variables of dynamical systems. For instance, the notations \(x(t),\mathbf{y(t)},u(t),\)\(\nu(t)\) and \(e(t)\) are rewritten as \(\mathbf{x},\mathbf{y},\mathbf{u},\mathbf{v}\) and \(e\).
As illustrated in Fig. 1, The central element of the proposed framework is a RNN, i.e., a recurrent network with a structure denoted by the adjacency matrix \(W^{r}\in\mathbb{R}^{n^{r}\times n^{r}}\). Here, the RNN consists of \(n^{r}\) neurons, for which the membrane potential dynamics are described as:

\[\dot{v}_{i}=\psi(v_{i})+I_{i} \tag{1}\]

\[I_{i}=W_{i}^{E}\partial(v^{E})+W_{i}^{r}\partial(v^{r}) \tag{2}\]

,where \(v=[v_{1}.v_{2}.\cdots.v_{n^{r}}]\in\mathbb{R}^{n^{r}}\) is the state or membrane potential of the RNN neurons, \(\partial(\cdot)\) is a Lipschitz nonlinear dendrite such as \(Sigmoid(\cdot).Tanh(\cdot)\), and, \(\psi(v)\colon\mathbb{R}^{n^{r}}\rightarrow\mathbb{R}^{n^{r}}\) is a Lipschitz leak-term where we used \(-\alpha_{l}v\), \(v^{E},v^{r}\) in non-spiking neurons are directly equal to input \(x\) and RNN state \(v\), and in spiking neurons \(v_{l}\) shows the filtered spike activity of neuron \(i\). \(S_{i}\left(t\right)\) is the spike train of the neuron \(i\) and modelled as a sum of Dirac delta-functions:

Figure 1: A schematic of the proposed architecture (plastic weights in red). (A) The feedforward pathway consists of an encoding layer with fix and random parameters, a RNN with trainable parameter and a decoder layer, which is trained in the pre-training process and stay constant during the RNN training process. (B) In the feedback pathway, the nonlinear optimal control is applied for estimating RNN parameters. It consists of actor and critic neural networks. (C) The bottom subplot shows the reference trajectories, and the top subplot shows the learned trajectories.
\[q_{i}=\big{(}S_{l}\,*\kappa\big{)}(t)=\int_{-\infty}^{t}S_{l}(s)\kappa(t-s)ds \tag{3}\]

\[\kappa(t)=\exp(-t/\tau)/\tau \tag{4}\]

Moreover, a linear decoder is assumed as:

\[\hat{y}=h(v)=W^{D}v \tag{5}\]

### Synaptic plasticity law

According to the equations mentioned in subsection 1.1, the RNN synaptic plasticity law can be modeled as a dynamical system. Consequently, principles of optimal control theory can be applied to derive a learning rule (optimal control law). For this purpose, we reformulate the AI tasks as a control problem. Thus, the output error were considered as follows:

\[e=\hat{y}-y\in\mathbb{R}^{c} \tag{6}\]

, where \(c\) is a number of classes, \(y\in\mathbb{R}^{c}\) denotes the desired output for the input pattern at time \(t\), and \(\hat{y}\in\mathbb{R}^{c}\) is the corresponding predicted output. By derivation of Eq. (6) with respect to \(t\), the error dynamics can be described as:

\[\dot{e}=\hat{y}-\hat{y}\ \in\mathbb{R}^{c} \tag{7}\]

Given that the decoder weights \(W^{D}\) are assumed constant during the RNN training process, consequently, the classification error dynamics only depend on the RNN neural dynamics. Rewriting the neural space equations (1) and (2) in the form of Eq. (7) gives:

\[\dot{e}=W^{D}\dot{v}\ -\dot{y}=W^{D}\big{(}\psi(v)+W^{E}\phi(v^{E})+W^{r} \phi(v^{r})\big{)}-\dot{y} \tag{8}\] \[=f(v)+g(v)u^{r}-f_{d}(v)-g_{d}(v)u_{d}^{r}\]

neural space Eq. (8) can be considered as the following affine sate space equation from:
\[\dot{e}=f(e)+g(e)u \tag{9}\]

Where the control input is considered as part of parameters of the RNN, which the number is selected according to the problem and training data:

\[u^{r}=[u^{f};u]=vec(W^{r})\in\mathbb{R}^{N^{r}} \tag{10}\]

, in which \(N^{r}=n^{r}\times n^{r}\) is the number of RNN parameters, \(vec(\cdot)\) is vectored form of a matrix, \(u^{f}\in\mathbb{R}^{N^{r}-N}\) is fixed RNN parameters, \(u\in\mathbb{R}^{N}\) is the RNN plastic synapse or the control input and \([u^{f};u]\) is combination of \(u^{f}\)and \(u\). Therefore, we have the following nonlinear continuous-time equations:

\[\begin{split}& f_{1}(v)=W^{D}\left(W^{E}\partial(v^{E})+\psi(v)+W^ {f}\partial(v^{f})\right)=f^{E}(v)+f^{f}(v)\\ & f^{E}(v)=W^{D}W^{E}\partial(v^{E})\\ & f^{f}(v)=W^{D}(\psi(v)+W^{f}\partial(v^{f}))\end{split} \tag{11}\]

\[g(v)=W^{D}\partial(v) \tag{12}\]

, where \(f(e)\colon\mathbb{R}^{c}\to\mathbb{R}^{c}\) is the drift dynamic of the system, and \(g(e)\colon\mathbb{R}^{c}\to\mathbb{R}^{c\times N}\) is the input dynamic of the system. A common and recommended action in this regard is to add noise into \(u\) and \(g(e)\) to realize the Persistence of Excitation (PE) condition [1] to collect sufficient information about the unknown parameters and make \(g(v)\) full rank.

In all tasks of AI, the goal of learning is to adapt the RNN's parameters \(W^{r}\) such that the error in each timestamp of the input pattern is minimized and the RNN's parameters or control input remains bounded. Thus, the following cost function was defined:

\[\min_{u(\cdot)\in u}\Im\big{(}u(\cdot).e(\cdot)\big{)}=\int_{t}^{t_{f}}\ell \big{(}e(\tau).u(\tau)\big{)}d\tau \tag{13}\]

\[\ell(e.u)=\eta e^{T}e+u^{T}Ru \tag{14}\]

, wherein \(\ell(\cdot\cdot)\) is the utility function, \(t_{f}\) is the last element of input patterns, \(R\) symmetric positive definite matrix and \(\eta\) is a constant positive hyper-parameter for
ensuring that the error in cost function is sufficiently affective. Hence, the optimal value function can be written:

\[V(e)=\underset{u(\cdot)\in U}{Inf}\Im(u.e) \tag{15}\]

Under typical assumptions, \(f(0)=0\) and \(f(v)+g(v)u\) is Lipschitz continues on a set \(\Omega\,\epsilon\,\mathbb{R}^{n}\) that contains the origin. Ultimately, it is desirable to achieve an optimal input control (weight update law) \(u^{*}\) that stabilizes the system Eq. (11) and minimizes the cost function Eq. (15). This kind of input control \(u\) is called admissible control [2].

Now, the learning rule has been formulated given the error dynamic in Eq. (11) and the cost function Eq. (15). To solve this dynamic optimization problem, the HJB equation is utilized, so the Hamiltonian of the cost function Eq. (15) associated with control input \(u\) is defined as:

\[\mathcal{H}(e.u.V_{e}\ )=\ell(e.u)+V_{e}^{T}(\dot{e}) \tag{16}\]

, where \(V_{e}=\partial V/\partial e\) is the partial derivative of the cost function, for admissible control policy \(\mu\) we have:

\[\mathcal{H}(e.\mu.V_{e}\ )=0 \tag{17}\]

The present study assumed that the solution to Eq. (17) is smooth giving the optimal cost function:

\[V^{*}(e)=\min_{u}(\int_{t}^{t_{f}}\ell\big{(}u(\tau).e(\tau)\big{)}d\tau) \tag{18}\]

, which satisfies the HJB equation

\[\underset{u}{min}\ \mathcal{H}(e.u.V_{e}^{*}\ )=0 \tag{19}\]

Assuming that the minimum on the left hand side Eq. (19) exists then by applying stationary condition \(\partial\mathcal{H}\left(e.u.V_{e}\ )/\partial\,u=0\), the learning rule (optimal control) can be obtained as:
\[u^{*}(e)=-\frac{1}{2}R^{-1}g^{T}(e)\,V_{e}^{*} \tag{20}\]

The optimal value function can be obtained as:

\[V^{*}(e)=\min_{u}(\int_{t}^{t_{f}}\eta e^{T}e+{u^{*}}^{T}Ru^{*}d\tau) \tag{21}\]

Inserting this optimal learning rule Eq. (20) into nonlinear Lyapunov equation Eq.(16) gives the formulation of the HJB equation Eq. (19) in terms of \(V_{e}^{*}\)

\[0=\eta e^{T}e+{V_{e}^{*}}^{T}(e)f(v)-\frac{1}{4}{V_{e}^{*}}^{T}(e)g(e)R^{-1}g^ {T}(v)V_{e}^{*}(e) \tag{22}\]

Finding the learning rule for the RNN requires solving the HJB equation for the value function and then substituting the solution to obtain the desired learning rule. Although HJB gives the necessary and sufficient condition for optimality of a learning rule (control law) with respect to a loss function; Unfortunately, due to the nonlinear characteristics of the RNN, solving the HJB equation in explicit form is difficult or even impossible to derive for systems of interest in practice. Thus, the proposed framework focuses on the ADP method to approximate its solution. Thus, we will focus on the ADP method to approximate its solution. To address this issue, with the inspiration of the dopaminergic region structure and conformity with the Weierstrass high-order approximation theorem [3], we proposed a novel actor-critic based on the s-RC. Succinctly, the objective of tuning the critic weights is to minimize the Bellman equation error, and the objective of tuning the actor weights is to minimize the approximate value.

#### Critic network design

In this subsection, a s-RC is exploited as critic NN to approximate the derivatives of the value function as follows:

\[\begin{array}{l}V_{e}\left(e\right)=W_{c}^{D}\,z_{c}+\varepsilon_{c}\in \mathbb{R}^{c\times 1}\\ z_{c}\in\mathbb{R}^{n_{c}\times 1}\\ W_{c}^{D}\in\mathbb{R}^{c\times n_{c}}\end{array} \tag{23}\]
Where \(z_{c}\) is the new representation of input \(e\), which is generated by the RNN and \(W_{c}^{D}\) is the decoder weight matrix.

As far as the weights of critic decoder provide the best approximate solution for the Hamiltonian function Eq. (16) are unknown. So, it estimated with \(\widehat{W}_{c}^{D}\):

\[\widehat{V}_{e}(e)=\widehat{W}_{c}^{D}z_{c} \tag{24}\]

It is desired to select the weights of the critic network to minimize the loss function of the critic is defined as follows:

\[E_{c}\big{(}\widehat{W}_{c}^{D}\big{)}=\frac{1}{2}e_{c}^{T}e_{c} \tag{25}\]

Where for given any admissible policy \(u\) the residual error:

\[\mathcal{H}(e.u.V_{e})=\mathcal{H}\big{(}e.u.\widehat{W}_{c}^{D}\big{)}=\eta e ^{T}e+u^{T}Ru+(\widehat{W}_{c}^{D}z_{c})^{T}(\hat{e})=e_{c} \tag{26}\]

So, \(\widehat{W}_{c}^{D}\to W_{c}^{D}\) and \(e_{c}\rightarrow\varepsilon_{c}\).

The weight update law for the critic weights is gradient descent algorithm, that is:

\[\begin{split}\hat{W}_{c}^{D}=-\alpha_{c}\frac{\partial E_{c}}{ \partial\widehat{W}_{c}^{D}}&=-\alpha_{c}(f(e)+g(e)u){z_{c}}^{T} e_{c}\\ &=-\alpha_{c}\sigma_{c}({\sigma_{c}}^{T}\widehat{W}_{c}^{D}+\eta e ^{T}e+u^{T}Ru)\end{split} \tag{27}\]

Where \(\alpha_{c}>0\) is learning rate and \(\sigma_{c}=(f(e)+g(e)u){z_{c}}^{T}\).

#### Actor network design

Similar to the critic, another s-RC is used as the actor to approximate the synaptic plasticity rule (feedback control policy):

\[\begin{split}& u(e)=W_{a}^{D}\,z_{a}+\varepsilon_{a}\in\mathbb{R}^{N \times 1}\\ &{z_{a}\in\mathbb{R}^{n_{a}\times 1}}\\ & W_{a}^{D}\in\mathbb{R}^{N\times n_{a}}\end{split} \tag{28}\]

Let \(z_{a}\) be the new representation of input \(e\), \(\widehat{W}_{a}^{D}\) an estimation of unknown matrix \(W_{a}^{D}\) based on existing training data, so the feedback control policy can be expressed as:

\[\widehat{u}(e)=\widehat{W}_{a}^{D}z_{a} \tag{29}\]
The loss function for the actor is defined:

\[E_{a}\big{(}\widehat{W}_{a}^{D}\big{)}=\frac{1}{2}e_{a}^{T}e_{a} \tag{30}\]

Where \(e_{a}\) is define to be the difference between Eq. (29) and Eq. (20)

\[e_{a}=\hat{u}-u=\widehat{W}_{a}^{D}z_{a}+\frac{1}{2}R^{-1}g^{T}(e)(\widehat{W}_ {c}^{D}z_{c}) \tag{31}\]

By applying the gradient descent, a weight update expression for the actor can be written as follows:

\[\hat{\bar{W}}_{a}^{D}=-\alpha_{a}\frac{\partial E_{a}}{\partial\widehat{W}_{a }^{D}}=-\alpha_{a}(\widehat{W}_{a}^{D}z_{a}+\frac{1}{2}R^{-1}g^{T}(e)(\widehat{ W}_{c}^{D}z_{c})){z_{a}}^{T} \tag{32}\]

Where \(\alpha_{a}>0\) is learning rate.

Finally, the RNN parameters update rule can be derived as follows:

\[vec(\widehat{W}^{r})=\hat{u}(e) \tag{33}\]

This completes the RNN learning rule.

During the RNN learning process, the feedback pathway via actor-critic NNs forces the output of the network (estimated class) to follow the reference trajectory (true class), an effect that is widely used in control theory. After a sufficiently long learning time, the feedforward pathway without feedback can classify input patterns with acceptable accuracy.

### The Analysis of stability and convergence

In this subsection, the stability and convergence of the entire proposed framework and classification error will be analyzed. For this purpose, we need to show \(\lim\limits_{t\to\infty}||\widehat{W}_{c}^{D}||=\lim\limits_{n_{e}\to \infty}||\widehat{W}_{c}^{D}||\to 0\), \(\lim\limits_{t\to\infty}||\widehat{W}_{a}^{D}||=\lim\limits_{n_{e}\to\infty}|| \widehat{W}_{a}^{D}||\to 0\) and \(\lim\limits_{t\to\infty}||e||=\lim\limits_{n_{e}\to\infty}||e||\to 0\). In offline problems, \(t=n_{e}n_{l}(n_{s}-1)+i\) where \(n_{e}\) is the number of the epochs, \(n_{s}\) is the number of samples, \(n_{l}\) is the time series length, and \(i\) is the index of the current time series element. Since in offline problems the training dataset \(n_{s}n_{l}\) is fixed so \(t\to\infty\) as \(n_{e}\to\infty\).
To prove asymptotically converge of the proposed framework, firstly, we need to show that the classification error \(e\) and the RNN critic and actor decoder weight estimation errors \(\widetilde{W}_{c}^{D}\), \(\widetilde{W}_{a}^{D}\) are uniformly ultimately bounded (UUB). Therefore, we consider the following Lyapunov and, to facilitate the UUB analysis, the following assumption is made, which can reasonably be satisfied under the current problem settings.

**Assumption 1** Assume the following conditions:

* We assumed \(g(\cdot)\) is bounded to positive constants i.e., \(\underline{g}\leq\|g(\chi)\|\leq\overline{g}\) therefore we have \(\underline{G}\leq G\leq\overline{G}\) where \(G=gR^{-1}g^{T}\).
* The ideal weights of the critic and actor RNN decoder are upper bounded so that \(\|W_{c}^{D}\|\leq\overline{W_{c}^{D}}\) and \(\|W_{a}^{D}\|\leq\overline{W_{a}^{D}}\).
* The approximation errors are upper bounded i.e., \(\|\varepsilon_{c}\|\leq\overline{\varepsilon_{c}}\), \(\|\varepsilon_{a}\|\leq\overline{\varepsilon_{a}}\).
* The outputs of the RNNs critic, actor are bounded i.e., \(\underline{z_{c}}\leq\|z_{c}\|\leq\overline{z_{c}}\), \(\underline{z_{a}}\leq\|z_{a}\|\leq\overline{z_{a}}\).

According to PE assumption \(\sigma_{c}=(f+gu){z_{c}}^{T}\) is bounded i.e., \(\underline{\sigma_{c}}\leq||\sigma_{c}||\leq\overline{\sigma_{c}}\).

Now, we consider the following Lyapunov function:

\[\begin{split} L(t)=L_{\widetilde{W}_{c}^{D}}(t)&+L_ {\widetilde{W}_{a}^{D}}(t)+L_{e}(t)\\ &=\frac{1}{2\alpha_{c}}\,tr\left\{\widetilde{W_{c}}^{D}{}^{T} \widetilde{W}_{c}^{D}\right\}+\frac{\alpha_{c}}{2\alpha_{a}}\,tr\left\{ \widetilde{W_{a}}^{D}{}^{T}\widetilde{W}_{a}^{D}\right\}\\ &+\alpha_{c}\alpha_{a}(e^{T}e+V(e))\end{split} \tag{34}\]

Where \(\alpha>0\), and \(\widetilde{W}_{c}^{D}\) is the weight estimation error of the critic decoder, which is defined as:

\[\widetilde{W}_{c}^{D}=\widetilde{W}_{c}^{D}-W_{c}^{D} \tag{35}\]

In addition, the dynamics of \(\widetilde{W}_{c}^{D}\) is expressed as:

\[\widetilde{W}_{c}^{D}=-\alpha_{c}(\sigma_{c}\sigma_{c}{}^{T}\widetilde{W}_{c} ^{D}+\sigma_{c}\varepsilon_{H}) \tag{36}\]

Where
\[\varepsilon_{H}=-\varepsilon_{c}(f+gu) \tag{37}\]

Moreover, the actor weight estimation error is define as:

\[\tilde{W}_{a}^{D}=\tilde{W}_{a}^{D}-W_{a}^{D} \tag{38}\]

Combining (23) and (28) with (32) yields:

\[\begin{split}\tilde{W}_{a}^{D}&=-\alpha_{a}\bigg{(} \tilde{W}_{a}^{D}z_{a}+\varepsilon_{a}+\frac{1}{2}R^{-1}g^{T}(e)\big{(}\tilde{W} _{c}^{D}z_{c}+\varepsilon_{c}\big{)}\bigg{)}{z_{a}}^{T}\\ &\hskip 142.26378pt=-\alpha_{a}\Big{(}\tilde{W}_{a}^{D}z_{a}+ \frac{1}{2}R^{-1}g^{T}(v)\big{(}\tilde{W}_{c}^{D}z_{c}\big{)}+\varepsilon^{ \prime}_{a}\Big{)}{z_{a}}^{T}\end{split} \tag{39}\]

Which \(\varepsilon^{\prime}_{a}\) becomes:

\[\varepsilon^{\prime}_{a}=-(\varepsilon_{a}+\frac{1}{2}R^{-1}g^{T}(e) \varepsilon_{c}) \tag{40}\]

Thus, the time derivative of \(L(t)\) is:

\[\begin{split} L(t)&=\dot{L}_{\tilde{W}_{c}^{D}}(t) +\dot{L}_{\tilde{W}_{a}^{D}}(t)+\dot{L}_{e}(t)\\ &\hskip 142.26378pt=\frac{1}{\alpha_{c}}tr\left\{\tilde{W}_{c}^{D }{}^{T}\dot{\tilde{W}}_{c}^{D}\right\}+\frac{\alpha_{c}}{\alpha_{a}}\,tr \left\{\tilde{W}_{a}^{D}{}^{T}\dot{\tilde{W}}_{a}^{D}\right\}+\alpha_{c} \alpha_{a}(2e^{T}\dot{e}\\ &\hskip 142.26378pt+\dot{V}(e))\end{split} \tag{41}\]

Substituting Eq. (36) into \(\dot{\tilde{W}}_{c}^{D}\) in Eq. (41), and assume \(\left\|\sigma_{c}{}^{T}\tilde{W}_{c}\right\|>\varepsilon_{H}\) the \(\dot{L}_{\tilde{W}_{c}^{D}}\) becomes:

\[\begin{split}\dot{L}_{\tilde{W}_{c}^{D}}(t)&=\frac {1}{\alpha_{c}}tr\left\{\tilde{W}_{c}^{D}{}^{T}\dot{\tilde{W}}_{c}^{D}\right\} \\ &\hskip 142.26378pt=-\frac{1}{\alpha_{c}}\,tr\left\{\alpha_{c} \tilde{W}_{c}^{D}{}^{T}\sigma_{c}\sigma_{c}{}^{T}\tilde{W}_{c}^{D}\right\}- \frac{1}{\alpha_{c}}\,tr\left\{\alpha_{c}\tilde{W}_{c}^{D}{}^{T}\sigma_{c} \varepsilon_{H}\right\}\\ &\hskip 142.26378pt\leq-(\sigma_{c}{}^{2})\left\|\tilde{W}_{c}^{ D}\right\|^{2}+\varepsilon_{H}{}^{2}\end{split} \tag{42}\]

Substituting Eq. (39) into \(\dot{\tilde{W}}_{a}^{D}\) in Eq. (41), the second term becomes:

\[\begin{split}\dot{L}_{\tilde{W}_{a}^{D}}(t)&=\frac {\alpha_{c}}{\alpha_{a}}\,tr\left\{\tilde{W}_{a}^{D}{}^{T}\tilde{W}_{a}^{D} \right\}\\ &\hskip 142.26378pt=-\frac{\alpha_{c}}{\alpha_{a}}\,tr\left\{ \tilde{W}_{a}^{D}{}^{T}\left[\alpha_{a}\,\Big{(}\tilde{W}_{a}^{D}z_{a}+\frac{1} {2}R^{-1}g^{T}(e)\big{(}\tilde{W}_{c}^{D}z_{c}\big{)}\right.\right.\\ &\hskip 142.26378pt+\left.\left.\varepsilon^{\prime}_{a}\Big{)}{ z_{a}}^{T}\right]\right\}\end{split} \tag{43}\]
According to Eq. (20), Eq. (29), and \(\ \widehat{W}_{a}^{D}z_{a}=-\frac{1}{2}R^{-1}g^{T}(e)\big{(}\widehat{W}_{c}^{D}z_{c} \big{)}\),we have:

\[L_{\widehat{W}_{a}^{D}}(t)\leq-(\alpha_{c}\underline{z_{a}}^{2})\left\|\widehat {W}_{a}^{D}\right\|^{2}+\frac{\alpha_{c}}{4\alpha_{a}}\left\|R^{-1}\right\|^{2 }\overline{g}^{2}\overline{z}_{c}^{2}\left\|\widehat{W}_{c}^{D}\right\|^{2}+ \frac{\alpha_{c}}{\alpha_{a}}\varepsilon^{\prime}_{a}^{2} \tag{44}\]

Substituting Eq. (29) into the error dynamics Eq. (11), and based on Assumption 1 a., third term of Eq. (41) become:

\[L_{e}(t)=\alpha_{c}\alpha_{a}(2e^{T}\dot{e}+\dot{V}(e))=2\alpha_{c}\alpha_{a}e ^{T}\dot{e}+2\alpha_{c}\alpha_{a}V_{e}^{T}\dot{e}\]

Substituting \(\dot{e}\) with Eq. (9) and \(V_{e}^{T}\dot{e}\) with \(-(\eta e^{T}e+u^{T}Ru)\) and according to Eq. (16) and Eq. (17):

\[\dot{L}_{e}(t)=2\alpha_{c}\alpha_{a}e^{T}(f(e)+g(e)u)-\alpha_{c}\alpha_{a}(\eta e ^{T}e+u^{T}u)\]

Substituting \(u\) with Eq. (20) and Eq.(24) which will be equal to \(u=-\frac{1}{2}R^{-1}g^{T}(v)\ \widehat{W}_{c}^{D}z_{c}\):

\[\dot{L}_{e}(t)=2\alpha_{c}\alpha_{a}e^{T}\left(f(e)-\frac{1}{2}g( e)R^{-1}g^{T}(e)\ \widehat{W}_{c}^{D}z_{c}\right)\] (45) \[\ \
\[\dot{L}_{e}(t)\leq\alpha_{c}\alpha_{a}\|e\|^{2}+\alpha_{c}\alpha_{a}\|f(e )\|^{2}\|e\|+\alpha_{c}\alpha_{a}\overline{Gz_{c}W_{c}^{D}}\ \|e\|\] \[\qquad\qquad\qquad\qquad-\alpha_{c}\alpha_{a}\eta\|e\|^{2}-\alpha_ {c}\alpha_{a}\lambda_{min}(R)\|u\|^{2}\] \[\qquad\qquad\qquad\leq-\alpha_{c}\alpha_{a}(\eta-1)\|e\|^{2}\] \[\qquad\qquad\qquad\qquad+\alpha_{c}\alpha_{a}\left(\overline{Gz_ {c}W_{c}^{D}}\ +\|f(e)\|^{2}\right)\|e\|\] \[\qquad\qquad\qquad\qquad-\alpha_{c}\alpha_{a}\lambda_{min}(R)\|u \|^{2}\]

Where \(\lambda_{min}(R)\) is the minimum eigenvalue of \(R\). Combining Eq. (42), Eq. (43) and Eq. (45):

\[\dot{L}(t)\leq-K_{\widetilde{W}_{c}^{D}}{}^{2}\big{\|}\widetilde {W}_{c}^{D}\big{\|}^{2}-K_{\widetilde{W}_{a}^{D}}{}^{2}\big{\|}\widetilde{W}_ {a}^{D}\big{\|}^{2}-K_{u^{2}}\|u\|^{2}-K_{e^{2}}\|e\|^{2}+K_{e}\|e\|\] \[\qquad\qquad\qquad\qquad+K_{1}\] \[\qquad\qquad\qquad\qquad-K_{e^{2}}\left(\|e\|-\frac{K_{e}}{2K_{ e^{2}}}\right)^{2}+\frac{K_{e}{}^{2}}{4K_{e^{2}}}+K_{1}\] \[\qquad\qquad\qquad\qquad\leq-K_{\widetilde{W}_{c}^{D}}{}^{2} \big{\|}\widetilde{W}_{c}^{D}\big{\|}^{2}-K_{\widetilde{W}_{a}^{D}}{}^{2} \big{\|}\widetilde{W}_{a}^{D}\big{\|}^{2}-K_{u^{2}}\|u\|^{2}\] \[\qquad\qquad\qquad\qquad-K_{e^{2}}\left(\|e\|-\frac{K_{e}}{2K_{ e^{2}}}\right)^{2}+\overline{K}\]

_Where_

\[K_{\widetilde{W}_{c}^{D}}{}^{2}=\frac{\sigma_{c}}{4\alpha_{a}} \|R^{-1}\|^{2}\overline{g}^{2}\overline{z_{c}}{}^{2} \tag{46}\] \[K_{\widetilde{W}_{a}^{D}}{}^{2}=\alpha_{c}z_{a}{}^{2}\] \[K_{e^{2}}=\alpha_{c}\alpha_{a}(\eta-1)\] \[K_{e}=\alpha_{c}\alpha_{a}\left(\overline{Gz_{c}}\overline{W_{c} ^{D}}\ +\|f(e)\|^{2}\right)\] \[K_{u^{2}}=\alpha_{c}\alpha_{a}\lambda_{min}(R)\] \[K_{1}=\varepsilon_{H}{}^{2}+\frac{\varepsilon^{\prime}\ a}{2} \leq\overline{K_{1}}\] \[K=\frac{K_{e}{}^{2}}{4K_{e^{2}}}+K_{1}\leq\overline{K}\]
Thus, \(L(t)<0\) if the proposed model hyper-parameters include \(\alpha_{c}\). \(\alpha_{a}\), \(\eta\), R, \(\psi(v)\) leaky term or \(W^{f}\phi(q^{\;f})\) fixed part of RNN's parameters in Eq. (11) are selected to satisfy three following inequalities:

\[0<\alpha_{c}<\frac{4\alpha_{a}{\sigma_{c}}^{2}}{\|R^{-1}\|^{2}\overline{g}^{2} \overline{z_{c}}^{2}} \tag{47}\]

And

\[0<\alpha_{a} \tag{48}\]

And

\[\eta>1 \tag{49}\]

Since all terms of \(K\) are bounded consequently according to [4] the classification error, the critic and the actor errors are semi-globally uniformly ultimately bounded (SGUUB), which is absolutely enough to classification task. Nevertheless, for reaching more stable condition following inequalities could be hold:

\[\|e\|>\frac{K_{e}}{K_{e^{2}}}+\sqrt{\frac{K_{1}}{K_{e^{2}}}} \tag{50}\]

Or

\[\left\|\widetilde{W}_{c}^{\;D}\right\|>\sqrt{\frac{\overline{K}}{K_{\widetilde {W}_{c}}^{2}}} \tag{51}\]

Or

\[\left\|\widetilde{W}_{a}^{\;D}\right\|>\sqrt{\frac{\overline{K}}{K_{\widetilde {W}_{a}}^{2}}} \tag{52}\]

Or

\[\|u\|>\sqrt{\frac{\overline{K}}{K_{u}}} \tag{53}\]
Thus, according to the Lyapunov theory the classification error \(e\), the RNN critic and actor decoder weight estimation errors \(\widehat{W}_{c}^{D}\), \(\widehat{W}_{a}^{D}\) and RNN parameters \(u\) are UUB.

Our aforementioned analysis based on Lyapunov theory implies that \(L_{e}(t)\) is bounded and decreasing. Therefore, we have:

\[\begin{split}& L_{e}(t+T)=L_{e}(t)+\int_{t}^{t+T}\dot{L}_{e}(\tau )d\tau\leq\ L_{e}(t)-\gamma T\\ &\implies L_{e}(t)\leq L_{e}(t+T)\end{split} \tag{54}\]

Where \(\gamma>0\). By substituting \(T=n_{l}n_{s}\) which is training dataset length and according to the Lyapunov function properties [5] and summing both sides of Eq. (54) we have:

\[\begin{split}&\sum_{t=0}^{n_{l}n_{s}}L_{e}(t)\leq\sum_{t=0}^{n_{l}n _{s}}L_{e}(t+T)\end{split} \tag{55}\]

Consequently, Eq. (55) guarantee that the classification error in each epoch's is not more than the previous epochs. Thus, as the number of epochs increases, the classification training error tends to zero (i.e. \(\lim\limits_{t\rightarrow\infty}e(t)\to 0\)).

In the same way, we can prove that for \(L_{\widehat{W}_{c}^{D}}(t)\) and \(L_{\widehat{W}_{a}^{D}}(t)\). This completes the stability and convergence proof.