Based on What We Can Control Artificial Neural Networks
How can the stability and efficiency of Artificial Neural Networks (ANNs) be
ensured through a systematic analysis method? This paper seeks to address that
query. While numerous factors can influence the learning process of ANNs,
utilizing knowledge from control systems allows us to analyze its system
function and simulate system responses. Although the complexity of most ANNs is
extremely high, we still can analyze each factor (e.g., optimiser,
hyperparameters) by simulating their system response. This new method also can
potentially benefit the development of new optimiser and learning system,
especially when discerning which components adversely affect ANNs. Controlling
ANNs can benefit from the design of optimiser and learning system, as (1) all
optimisers act as controllers, (2) all learning systems operate as control
systems with inputs and outputs, and (3) the optimiser should match the
learning system. Please find codes:
\url{https://github.com/RandomUserName2023/Control-ANNs}.
Cheng Kang, Xujing Yao
# Based on What We Can Control Artificial Neural Networks

###### Abstract

How can the stability and efficiency of Artificial Neural Networks (ANNs) be ensured through a systematic analysis method? This paper seeks to address that query. While numerous factors can influence the learning process of ANNs, utilizing knowledge from control systems allows us to analyze its system function and simulate system responses. Although the complexity of most ANNs is extremely high, we still can analyze each factor (e.g., optimiser, hyperparameters) by simulating their system response. This new method also can potentially benefit the development of new optimiser and learning system, especially when discerning which components adversely affect ANNs. Controlling ANNs can benefit from the design of optimiser and learning system, as (1) all optimisers act as controllers, (2) all learning systems operate as control systems with inputs and outputs, and (3) the optimiser should match the learning system. Please find codes: [https://github.com/RandomUserName2023/Control-ANNs](https://github.com/RandomUserName2023/Control-ANNs).

Optimizer Controller Learning System Control System Fuzzy Logic Filter

## 1 Introduction

Controlling artificial neural networks (ANNs) has become an urgent issue on such a dramatically growing domain. Although ANN models, such as, vision models (e.g., CNN [21], VGG19 [33], ResNet50 [11], EfficientNet [34], ViT [7]), language models (e.g., BERT [6], GPT [29], PaLM [4]), and generative models (e.g., GAN [9], VAE [18], Stable Diffusion Models [15, 31]), all require input and output, as they aim to map the gap between their output and the desired output. However, basically, CNN-based vision models prefer SGDM [28] optimiser, and generative models tend to rely on AdaM optimiser. Using various architecture on CNN-based vision models (e.g., from VGG19 to ResNet50, from GAN to CycleGAN [40], and from CNN to FFNN [13]) yield significantly varied results for classification and generation tasks. Two critical questions arise: **(1)** why some of them satisfy the corresponding optimiser, **(2)** based on what to propose an advanced ANN architecture and a proper optimiser.

Compared to existing era-acrossing optimisers, such as SGD [30, 5, 38], SGDM [28, 25], AdaM [17, 3], PID [36], and Gaussian LPF-SGD [2], we proposed a FuzzyPID optimiser modified by fuzzy logic to avoid vibration during PID optimiser learning process. Referring to Gaussian LPF-SGD (GLFP-SGD), we also proposed two filter processed SGD methods according to the low and high frequency part during the SGD optimiser learning process: low-pass-filter SGD (LPF-SGD) and high-pass-filter SGD (HPF-SGD). To achieve stable and convergent performance, we simulate these above optimisers on the system response to analyze their attributes. When using simple and straightforward architecture (without high techniques, such as, BN [16], ReLU [26], pooling [37], and exponential or cosine decay [24]), we found their one step system response are always consistent with their training process. Therefore, we conclude that every optimiser actually can be considered as a controller that optimise the training process. Results using HPF-SGD indicate that the high frequency part using SGD optimiser significantly benefits the learning process and the classification performance.

To analyze the learning progress of most ANNs, for example, CNN using backpropagation algorithm, FFNN using forward-forward algorithm, and GAN such a generative model using random noise to generate samples. We assume
above three mentioned models here essentially can be represented by corresponding control systems. But the difficulty is that when using different optimisers, especially, AdaM, we cannot analyze its stability and convergence, as the complexity is extremely high. Thus, we use MATLAB Simulink to analyze their system response, as well as their generating response. Experiment results indicate that advanced architectures and designs of these three ANNs can improve the learning, such as residual connections (RSs) on ResNets, a higher Threshold on FFNN, and a cycle loss function on CycleGAN.

Based on the knowledge of control systems [27], designing proper optimisers (or controllers) and advanced learning systems can benefit the learning process and complete relevant tasks (e.g., classification and generation). In this paper, we design two advanced optimisers and analyze three learning systems relying on the control system knowledge. The contributions are as follows:

**Optimisers are controllers. (1)** PID and SGDM (PI controller) optimiser performs more stable than SGD (P controller), SGDM (PI controller), AdaM and fuzzyPID optimisers on most residual connection used CNN models. **(2)** HPF-SGD outperforms SGD and LPF-SGD, which indicates that high frequency part is significant during SGD learning process. **(3)** AdaM is an adaptive filter that combines an adaptive filter and an accumulation adaptive part.

**Learning systems of most ANNs are control systems. (1)** Most ANNs present perfect consistent performance with their system response. **(2)** We can use proper optimisers to control and improve the learning process of most ANNs.

**The Optimiser should match the learning system. (1)** RSs based vision models prefer SGDM, PID and fuzzyPID optimisers. **(2)** RS mechanism is similar to AdaM. particularly, SGDM optimizes the weight of models on the time dimension, and RS optimizes the model on the space dimension. **(3)** AdaM significantly benefits FFNN and GAN, but PID and FuzzyPID does CycleGAN most.

## 2 Problem Statement and Preliminaries

To make ANNs more effective and adaptive to specific tasks, controlling ANNs has become necessary. We initialize a parameter of a node in the ANN model as a scalar \(\theta_{0}\). After enough time of updates, the optimal value of \(\theta^{*}\) can be obtained. We simplify the parameter update in ANN optimisation as a one-step response (from \(\theta_{0}\) to \(\theta^{*}\) ) in the control system. The Laplace transform of \(\theta^{*}\) is \(\theta^{*}/s\). We denote the weight \(\theta(t)\) at iteration \(t\). The Laplace transform of \(\theta(t)\) is denoted as \(\theta(s)\), and that of error \(e(t)=\theta^{*}-\theta(t)\) as \(E(s)\):

\[E(s)=\frac{\theta^{*}}{s}-\theta(s) \tag{1}\]

Considering the collaboration of backward and forward algorithms, the Laplace transform of the training process is

\[U(s)=(\textit{Controller1}+\textit{Controller2}\cdot F(s))\cdot E(s) \tag{2}\]

\(F(s)\) is the forward system which has the capability to affect \(U(s)\) beforehand. In our case, \(u(t)\) corresponds to the update of \(\theta(t)\). _Controller1_ is the parameter update algorithm for the backward process, and _Controller2_ is the parameter update algorithm for the forward process. Therefore, we replace \(U(s)\) with \(\theta(s)\) and \(E(s)\) with \((\theta^{*}/s)-\theta(s)\). Equation 2 can be rewritten as

\[\theta(s)=(\textit{Controller1}+\textit{Controller2}\cdot F(s))\cdot\left( \frac{\theta^{*}}{s}-\theta(s)\right) \tag{3}\]

Finally, we simplify the formula of training a model as:

\[\theta(s)=\frac{\textit{Controller}}{\textit{Controller}+1}\cdot\frac{ \theta^{*}}{s} \tag{4}\]

where \(\textit{Controller}=\textit{Controller1}+\textit{Controller2}\cdot F(s)\). \(\theta^{*}\) denotes the optimal model which we should get at the end. Simplifying \(\theta(s)\) further as below:

\[\theta(s)=\textbf{Controller}(\mathbf{s})\cdot\mathbf{C}(\mathbf{s}) \tag{5}\]

Figure 1: The schematic structure of training ANN models. C(s) is the controller to train the target ANN model.
where \(\mathbf{Controller}(\mathbf{s})=\mathit{Controller}/(\mathit{Controller}+1)\), and \(\mathbf{C}(\mathbf{s})=\theta^{*}/s\). Based on above analytic thought, as shown in Figure 1 there are two ways to obtain an optimal \(\theta(s)\) and to make the training process better: **(1)** using a better **Controller** and **(2)** constructing a better training or control system \(\mathbf{C}(\mathbf{s})\).

## 3 Optimisers are Controllers

In this section, we review several widely used optimisers, such as SGD [30; 5; 38], SGDM [28; 25], AdaM [17; 3], PID-optimiser [36] and Gaussian LPF-SGD [2]. In the training process of most ANNs, there are diverse architectures used to satisfy various tasks. We analyze the performance of optimisers in terms of one node of backpropagation based ANN models. Please see the proof in Appendix A.

### AdaM Optimiser

AdaM [17] has been used to optimise the learning process of most ANNs, such as GAN, VAE, Transformer-based models, and their variants. We simplify the learning system of using AdaM on ANNs as below:

\[\theta(s)=\frac{K_{p}s+K_{i}}{Ms^{2}+(K_{p}-Mln\beta_{1})s+K_{i}}\cdot\frac{ \theta^{*}}{s} \tag{6}\]

where \(M\) is an adaption factor which will dynamically adjust the learning during the training process, and it can be derived from:

\[M=\frac{1}{\sqrt{\frac{\sum_{i=0}^{t}\beta_{2}^{t-i}(\partial L_{t}/\partial \theta_{i})^{2}}{\sum_{i=0}^{t}\beta_{2}^{t-1}}}+\epsilon}\cdot\frac{1}{\sum_ {i=0}^{t}\beta_{1}^{t-1}} \tag{7}\]

Apart from the adaption part \(M\), AdaM can be thought as the combination of SGDM and an adaptive filter with the cutoff frequency \(\omega_{c}=ln(\beta_{1})\).

### Filter Processed SGD optimiser

SGD learning process can be filtered under carefully designed filters. GLPF-SGD [2] used a low pass Gaussian-filter to smooth the training process, as well as actively searching the flat regions in the Deep Learning (DL) optimisation landscape. Eventually, we simplify the learning system of using SGD with filters on ANNs as below:

\[\theta(s)=\frac{Gain\cdot\prod_{i=0}^{m}{(s+h_{i})}}{Gain\cdot\prod_{i=0}^{m }{(s+h_{i})+\prod_{j=0}^{n}{(s+l_{j})}}}\cdot\frac{\theta^{*}}{s} \tag{8}\]

where designed \(Filter\) have the order, such as \(n\) for the low pass and \(m\) for the high pass (\(h_{i}\) is the coefficient of the high pass part and \(l_{i}\) is the coefficient of the low pass part), and \(Gain\) is the gain factor:

\[Filter=Gain\cdot\frac{(s+h_{0})(s+h_{1})...(s+h_{m})}{(s+l_{0})(s+l_{1})...(s+l _{n})} \tag{9}\]

### PID and FuzzyPID optimiser

Based on PID optimiser [36], we design a PID controller which is optimised by fuzzy logic to make the training process more stable while keeping the dominant attribute of models. For instance, the ability to resist the disturbance of the poisoned samples, the quick convergent speed and the competitive performance.

There are two key factors which affect the performance of the Fuzzy PID optimiser: (1) the selection of Fuzzy Universe Range \([-\varphi,\varphi]\) and (2) Membership Function Type \(f_{m}\).

\[\widehat{K}_{\mathrm{P,I,D}}=K_{\mathrm{P,I,D}}+\Delta K_{\mathrm{P,I,D}} \tag{10}\]

\[\Delta K_{\mathrm{P,I,D}}=Defuzzy(E(s),Ec(s))\cdot K_{\mathrm{P,I,D}} \tag{11}\]

\(Defuzzy(s)=f_{m}(round(-\varphi,\varphi,s))\)
where \(\Delta K_{\rm P,I,D}\) refer to the default gain coefficients of \(K_{\rm P}\), \(K_{\rm I}\) and \(K_{\rm D}\) before modification. \(E(s)\) is the back error, and \(Ec(s)\) is the difference product between the \(Laplace\) of \(e(t)\) and \(e(t-1)\). The Laplace function of this model \(\theta(s)\) eventually becomes:

\[\theta(s)=\frac{\widehat{K}_{d}s^{2}+\widehat{K}_{p}s+\widehat{K}_{i}}{ \widehat{K}_{d}s^{2}+(\widehat{K}_{p}+1)s+\widehat{K}_{i}}\cdot\frac{\theta^{ *}}{s} \tag{12}\]

where \(\widehat{K}_{p}\), \(\widehat{K}_{i}\) and \(\widehat{K}_{d}\) should be processed under the fuzzy logic. By carefully selecting the learning rate \(r\), \(\theta(s)\) becomes a stable system.

The PID [1] and Fuzzy PID [35] controllers have been used to control a feedback system by exploiting the present, past, and future information of prediction error. The advantages of a fuzzy PID controller includes that it can provide different response levels to non-linear variations in a system. At the same time, the fuzzy PID controller can function as well as a standard PID controller in a system where variation is predictable.

## 4 Control Systems of ANNs

In this section, to systematically analyze the learning process of ANNs, we introduce three main common-used control systems that we believe can be respectively connected to backpropagation based CNNs, forward-forward algorithm based FFNNs, and GANs: **(1)** backward control system, **(2)** forward control system using different hyperparameters, and **(3)** backward-forward control system on different optimisers and hyperparameters. Please see the proof in Appendix B.

### Backward Control System

Traditional CNNs use the backpropagation algorithm to update initialized weights, and based on errors or minibatched errors between real labels and predicted results, optimisers are used to control on how the weight should be updated. According to the deduction of PID optimiser [36], the training process of Deep Neural Networks (DNNs) can be conducted under a step response of control systems. However, most common-used optimisers have their limitations, such as **(1)** SGD costs a very long term to reach convergence, **(2)** SGDM also has the side effect of long term convergence even with the momentum accelerating the training, **(3)** AdaM presents a frequent vibration during the training because of the merging of momentum and root mean squared propagation (RMSprop), **(4)** PID optimiser has better stability and convergence speed, but the training process is still vibrating. This proposed fuzzyPID optimiser can keep the learning process more stable, because it can be weighted towards types of responses, which seems like an adaptive gain setting on a standard PID optimiser. Finally, we get the system function \(\theta(s)\) of ANNs by using FuzzyPID optimisers as an example below:

\[\theta(s)=\frac{\textit{FuzzyPID}}{\textit{FuzzyPID}+1}\cdot\frac{\theta^{ *}}{s} \tag{13}\]

### Forward-Forward Control System

The using of forward-forward computing algorithm was systematically analyzed in forward-forward neural network [13] which aims to track features and figure out how ANNs can extract them from the training data. The Forward-Forward algorithm is a greedy multilayer learning procedure inspired by Boltzmann machines [14] and noisy contrastive estimation [10]. To replace the forward-backward passes of backpropagation with two forward passes that operate on each other in exactly the same way, but on different data with opposite goals. In this system, the positive pass operates on the real data and adjusts the weights to increase the goodness in each hidden layer; the negative pass operates on the negative data and adjusts the weights to reduce the goodness in each hidden layer. According to the training process of FFNN, we get its system function \(\theta(s)\) as below:

\[\theta(s)=\left\{\left(-(1-\lambda)\frac{\theta^{*}}{s}+\lambda\frac{\theta^{ *}}{s}-\left[\theta(s)-\frac{Th}{s}\right]\right)\right\}\cdot Controller \tag{14}\]

where \(\lambda\in[0,1]\) is the portion of positive samples, and \(Th\) is the given Threshold according to the design [13]. Input should contain negative and positive samples, and by adjusting the Threshold \(Th\), the embedding space can be optimised. In each layer, weights should be updated on only corresponding errors that can be computed by subtracting the Threshold \(Th\). We finally simplify \(\theta(s)\) as:

\[\theta(s)=\frac{1}{Controller+1}\cdot\left(\frac{(2\lambda-1)\theta^{*}+Th}{s}\right) \tag{15}\]
Because \((2\lambda-1)\theta^{*}+Th\geq 0\), the system of FFNN is stable. Additionally, when \(\lambda=0.5\) and \(Th=1.0\), the learning system of FFNN (the second half part of Equation 15) will become to that of backpropagation based CNN, as we assume \(\theta^{*}\approx 1.0\). When \(\lambda=0.5\), the optimal result \(\theta^{*}\) has no relationship with the learning system.

### Backward-Forward Control System

GAN is designed to generate samples from the Gaussian noise. The performance of the GAN depends on its architecture [39]. The generative network uses random inputs to generate samples, and the discriminative network aims to classify whether the generated sample can be classified [9]. We get its \(\theta(s)\) as below:

\[\theta_{D}(s)=controller\cdot\theta_{G}(s)\cdot E(s) \tag{16}\]

\[\theta_{G}(s)=controller\cdot E(s) \tag{17}\]

\[E(s)=\frac{\theta_{D}^{*}}{s}-\theta_{D}(s) \tag{18}\]

where \(\theta_{D}(s)\) is the desired Discriminator, \(\theta_{G}(s)\) is the desired Generator. \(E(s)\) is the feed-back error. \(\theta_{G}^{*}\) is the optimal solution of the generator, and \(\theta_{D}^{*}\) is the optimal solution of the discriminator.

Eventually, we simplify \(\theta_{G}(s)\) and \(\theta_{D}(s)\) as below:

\[\theta_{G}(s)=\frac{1}{2}\cdot\left(\frac{\theta_{D}^{*}}{Controller}\pm \sqrt{(\frac{\theta_{D}^{*}}{Controller})^{2}-\frac{4}{s}}\right) \tag{19}\]

\[\theta_{D}(s)=\theta_{G}^{2}(s) \tag{20}\]

where if set \(\theta_{G}(s)=0\), we get one pole point \(s=0\). When using SGD as the \(controller\), \(\theta_{G}(s)\) is a marginally stable system.

## 5 Experiments

### Simulation

As we believe that the training process of most ANNs can be modeled as the source response of control systems, we use Simulink (MATLAB R2022a) to simulate their response to different sources. For the classification task, because all models aim to classify different categories, we set a step source as illustrated in [36]. For the sample generation task, to get a clear generating result, we use a sinusoidal source.

### Experiment Settings

We train our models on the MNIST [23], CIFAR10 [20], CIFAR100 [20] and TinyImageNet [22] datasets. For an apple-to-apple comparison, our training strategy is mostly adopted from PID optimiser [36] and FFNN [13]. To optimise the learning process, we **(1)** firstly use seven optimisers for the classification task on backpropagation algorithm based ANNs. **(2)** Secondly, we choose some important hyperparameters and simulate the learning process of FFNN. **(3)** Lastly, to improve the stability and convergence during the training of GAN, we analyze its system response on various optimisers. All models are trained on single Tesla V100 GPU. All the hyper-parameters are presented in Table 3 of Appendix E.

#### 5.2.1 Backward Control System

We design one neural network using backpropagation algorithm with \(2\) hidden layers, setting the learning rate \(r\) at \(0.02\) and the fuzzy universe range \(\varphi\) at \([-0.02,0.02]\). We initialize \(K_{P}\) as \(1\), \(K_{I}\) as \(5\), and \(K_{D}\) as \(100\). Thus, we compare seven different optimisers: SGD (P controller), SGDM (PI controller), AdaM (PI controller with an Adaptive Filter), PID (PID controller), LPF-SGD, HPF-SGD and FuzzyPID (fuzzy PID controller) on the above ANN model. We set Gaussian membership function as the default membership function. See filter coefficients in Table 4 of Appendix E. In Table 5 of Appendix E, there is a set of hyperparameters that we have used to train CIFAR10, CIFAR100 and TinyImageNet.

#### 5.2.2 Forward-Forward Control System

Following the forward-forward algorithm [13], we design one forward-forward neural network (FFNN) with \(4\) hidden layers each containing \(2000\) ReLUs and full connectivity between layers, by simultaneously feeding positive and negative samples into the model to teach it to distinguish the handwriting number (MNIST). We also carefully select the proportion of positive and negative samples. The length of every block is \(60\).
#### 5.2.3 Backward-Forward Control System

To demonstrate the relationship between the control system and the learning process of some complex ANNs, we choose the classical GAN [9]. Both the generator and the discriminator comprise \(4\) hidden layers. To verify the influence of different optimisers on GAN, we employ SGD, SGDM, AdaM, PID, LPF-SGD, HPF-SGD and fuzzyPID to generate the handwriting number (MNIST). We set the learning rate at \(0.0002\) and the total number of epochs at \(200\).

## 6 Results and Analysis

In this section, we present simulation performance, classification accuracy, error rate and generation result, using different optimisers and advanced control systems.

### Backward Control System on CNN

Before doing the classification task, we firstly simulate the step response of backpropagation based ANNs on each controller (optimiser). As observed in Figure 1(b) and Figure 1(c), AdaM optimiser can rapidly converge to the optimal but with an obvious vibration. Although FuzzyPID cannot rapidly converge to the optimal, there is no obvious vibration during the training. Other optimisers, such as HPF-SGD, SGDM and PID, perform lower than AdaM and FuzzyPID in terms of the training process. In Figure 1(a), the response of AdaM controller is faster than others, and FuzzyPID follows it. However, due to the overshoot on AdaM, the stability of ANN system when using the AdaM controller tends to be lower. This overshoot phenomenon is reflected on the training process of Adam optimising in Figure 1(b) and Figure 1(c).

We summarize the result of classifying MNIST in Table 1. Under the same condition, SGD optimiser reaches the testing accuracy at \(91.98\%\), but other optimisers can reach above \(97\%\). FuzzyPID gets the highest training and testing accuracy rates using Guassian membership function. In Figure 2, if considering the rise time, the settling time and the overshoot, the fuzzy optimiser outperforms other optimisers. A better optimiser (or controller) that has inherited advanced knowledge and sometimes has been effectively designed is beneficial for the classification performance.

### Forward Forward Control System on FFNN

We also simulate the control system of this proposed FFNN and compare its system response on different hyperparameters. In Figure 3, SGD controller still cannot reach the target, and AdaM controller reacts fastest approaching to the target. However, SGDM controller lags behind PID in terms of the step response. Because of the low frequency part of LPF-SGD, it climbs slower than HPF-SGD. Although the differential coefficient D of PID optimiser can help reduce overshoot and overcome oscillation and reduce the adjustment time, its performance cannot catch up with AdaM. Compared to Table 2, AdaM outperforms other optimisers in terms of error rates, and the performance of these seven optimisers are echoing Figure 2(a). A higher portion of positive samples can contribute to the classification, and a higher

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**optimiser** & SGD & SGDM & Adam & PID & LPF-SGD & HPF-SGD & FuzzyPID \\ \hline
**Training**\(Accuracy\) & \(91.48_{\pm 0.03}\) & \(97.78_{\pm 0.00}\) & \(99.46_{\pm 0.02}\) & \(99.45_{\pm 0.01}\) & \(11.03_{\pm 0.01}\) & \(93.35_{\pm 0.02}\) & \(99.73_{\pm 0.00}\) \\
**Testing**\(Accuracy\) & \(91.98_{\pm 0.05}\) & \(97.11_{\pm 0.02}\) & \(97.81_{\pm 0.10}\) & \(98.18_{\pm 0.02}\) & \(10.51_{\pm 0.03}\) & \(93.45_{\pm 0.09}\) & \(98.24_{\pm 0.10}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: The results of ANN based on the backpropogation algorithm on MNIST data. Using the 10-fold cross-validation, the average and standard variance results are shown below.

Figure 2: The step response, training curve and loss curve using different controllers, such as SGD, SGDM, AdaM, PID, LPF-SGD, HPF-SGD and FuzzyPID optimisers.
\(Threshold\) can benefit more. For the step response in Figure (c)c, although AdaM ( \(Threshold=0.5\), \(portion\) of positive samples is \(70\%\), and \(portion\) of negative samples is \(30\%\)) and AdaM ( \(Threshold=0.5\), \(portion\) of positive samples is \(50\%\), and \(portion\) of negative samples is \(50\%\)) rise fateset, the final results in Table 2 present that AdaM ( \(Threshold=5.0\), \(portion\) of positive samples is \(50\%\), and \(portion\) of negative samples is \(50\%\)) get a lower error rate.

### Backward-Forward Control System on GAN

For the sample generation task, we also simulate the system response of GANs on each controllers (optimisers) and summarize the result in Figure 5. Apart from AdaM, LPF-SGD and HPF-SGD, all controllers have obvious noise, and interestingly, this phenomenon can be seen in Figure 4. The generated MNIST using Adam optimiser has no noise and can be easily recognized, and not surprised, the source response of AdaM in Figure 5 can finally converge. Figure 4 and Figure 5 mutually echo each other. Eventually, when using classical GAN to generate samples, AdaM should be the best optimiser to optimise the update of weights. The generated MNIST sample sometimes cannot be recognized, and GAN generates only same samples. One reason for this can be observed in Figure 5, where the sinusoidal signals generated by these four controllers, such as PID, LPF-SGD, HPF-SGD and FuzzyPID move up and down, potentially leading to an unstable and same generation output.
## 7 Discussion

### Why various optimisers are controllers during the learning process?

Under the same training condition (e.g., same architecture and hyperparameters), corresponding optimisers can tackle with specific tasks. Residual connection used vision models prefer SGDM, HPF-SGD and PID optimisers (Seen from Figure 14 of Appendix F). There is an obvious overshoot on the step response of AdaM controller (Seen from Figure 10), and a similar vibration can be found in the testing curve of Figure 14 of Appendix F. The classification task always needs a rapid response to save learning resources, but if stability and robustness are the priorities, we should set others as the optimizer, such as PID or FuzzyPID optimiser, which under fuzzy logic adjustment, demonstrates a superior step response (can be seen from Figure 2a). Moreover, for the generation task, GAN satisfies AdaM optimiser. We found that the adaptive part of AdaM can rapidly adjust the learning process. However, other optimisers, such as SGD, SGDM and PID, generate samples with obvious noise and output the same samples make the generated sample cannot be recognized easily (can be seen from Figure 4 and Figure 5). For particular needs (e.g., Image-to-Image Translation), CycelGAN, this advanced generation system was proposed to generate samples from one data pool and to improve its domain adaption on the target data pool. Coincidentally, we found that CycleGAN has a preference for the PID optimiser. Therefore, it is necessary to design a stable and task-satisfied optimiser on a specificly designed learning system. However, given that the system functions of most learning systems are extremely complex, simulating their system responses has become a viable way to analyze them. We conclude that to achieve best performance, every ANN should use the proper optimiser according to its learning system.

### How various learning systems can be analyzed?

Numerous advanced components have enhanced ANNs. Conducting a quantitative analysis on each of them can pave the way for the development of new optimisers and learning systems. For the classification task using a backward control system, in one node of the learning system, and in terms of analyzing a single component, the rise time, peak time, overshoot (vibration), and settling time [36; 27] can be the metrics to evaluate the performance of such component on learning systems. To visualize the learning process, FFNN was proposed by [13], and effectively, this forward-forward-based training system also can achieve competitive performance compared to backpropagation-based models. The \(Threshold\) - one hyperparameter - can significantly benefit the convergence speed, as it has the effect of proportional adjustment (same as a stronger P in PID controller). The portion of positive samples can slightly affect the classification result, as because the proportional adjustment is too weak on FFNN learning system (Seen from Equation 15). Additionally, the system response on various sources can also serve as a metric to evaluate the learning system. We conclude that there are two main branches to improve ANNs: **(1)** develop a proper optimiser; **(2)** design a better learning system. On the one hand, for example, the system response of GAN has high-frequency noise and cannot converge using SGD, SGDM and PID optimisers (seen from Figure 5). One possible solution is adding an adaptive filter. Thus, AdaM outperforms other optimisers on generating samples (Seen from Figure 4). The overshoot of AdaM and SGDM during the learning process of classification tasks can accelerate the convergence, but its side-effect of vibration brings us to PID and FuzzyPID. Therefore, developing a task-matched optimizer according to the system response determines the final performance of ANNs. On the other hand, to satisfy various task requirements, learning systems also should become stable and fast. For example, \(\theta_{G}(s)\) has two system functions as derived from Eq 19), to offset the side effect by considering the possible way using extra generator. That can explain why other advanced GANs using multi-generators (e.g., CycleGAN) can generate high-quality samples than the classical GAN.

## 8 Limitations

Although we systematically proved that **(1)** the optimiser acts as a controller and **(2)** the learning system functions as a control system, in this preliminary work, there are three obvious limitations: **a.** we cannot analyze larger models due to the complexity introduced by advanced techniques; **b.** the system response of some ANNs (e.g., FFNN) may not perfectly align with their real performance; **c.** we cannot always derive the solution of complex learning system.

## 9 Conclusion

In this study, we showed comprehensive empirical study investigating the connection between control systems and various learning systems of ANNs. We provided a systematic analysis method for several ANNs, such as CNN, FFNN, GAN, CycleGAN, and ResNet on several optimisers: SGD, SGDM, AdaM, PID, LPF-SGD, HPF-SGD and FuzzyPID. By analyzing the system response of ANNs, we explained the rationale behind choosing appropriate optimisers for different ANNs. Moreover, designing better learning systems under the use of proper optimiser can satisfy task
requirements. In our future work, we will intend to delve into the the control system of other ANNs, such as Variational Autoencoders (VAEs), diffusion models, Transformer-based models and so on, aw well as the development of optimisers, as we believe the principles of control systems can guide improvements in all ANNs and optimisers.
