A critical look at deep neural network for dynamic system modeling
Neural network models become increasingly popular as dynamic modeling tools
in the control community. They have many appealing features including nonlinear
structures, being able to approximate any functions. While most researchers
hold optimistic attitudes towards such models, this paper questions the
capability of (deep) neural networks for the modeling of dynamic systems using
input-output data. For the identification of linear time-invariant (LTI)
dynamic systems, two representative neural network models, Long Short-Term
Memory (LSTM) and Cascade Foward Neural Network (CFNN) are compared to the
standard Prediction Error Method (PEM) of system identification. In the
comparison, four essential aspects of system identification are considered,
then several possible defects and neglected issues of neural network based
modeling are pointed out. Detailed simulation studies are performed to verify
these defects: for the LTI system, both LSTM and CFNN fail to deliver
consistent models even in noise-free cases; and they give worse results than
PEM in noisy cases.
Jinming Zhou, Yucai Zhu
# A critical look at deep neural network for dynamic system modeling

###### Abstract

Neural network models become increasingly popular as dynamic modeling tools in the control community. They have many appealing features including nonlinear structures, being able to approximate any functions. While most researchers hold optimistic attitudes towards such models, this paper questions the capability of (deep) neural networks for the modeling of dynamic systems using input-output data. For the identification of linear time-invariant (LTI) dynamic systems, two representative neural network models, Long Short-Term Memory (LSTM) and Cascade Forward Neural Network (CFNN) are compared to the standard Prediction Error Method (PEM) of system identification. In the comparison, four essential aspects of system identification are considered, then several possible defects and neglected issues of neural network based modeling are pointed out. Detailed simulation studies are performed to verify these defects: for the LTI system, both LSTM and CFNN fail to deliver consistent models even in noise-free cases; and they give worse results than PEM in noisy cases.

keywords: Deep neural network, Model structure, Error criteria, Consistency, Model validation, System identification.

## 1 Introduction

Process modeling is fundamental to most process applications, from control, optimization to fault diagnosis and soft sensor. An accurate model that well reflects the process behavior is essential for success of all these applications. Neural network and deep learning models have become increasingly popular in the control community, inspired by their tremendous success in, e.g., Computer Vision (CV), Natural Language Processing (NLP) [1; 2]. Unlike system identification theory that starts from the well-established theory for linear systems [3; 4], neural networks have inherently nonlinear structures. Moreover, they are so-called universal approximators capable of approximating any functions within any degree of accuracy [5; 6]. Numerous papers have been published for the development of neural network based methods for various applications in systems and control [7; 8; 9].

The input and output spaces of a neural network are very general: the input could be signals, images, etc., while the output could be numerical values or classifications. Among the many possibilities, this paper only focuses on the case where the neural network is used to learn the dynamic relations between process inputs and outputs. Such model can be used, e.g., as the internal model in model predictive control (MPC), for residual generation in FDI, or as a soft sensor.

According to some state-of-the-art review papers [7; 8; 9; 10; 11], three representative network structures are Feedfoward Neural Network (FNN), Recurrent Neural Network (RNN), and Convolution Neural Network (CNN). FNN is a classical structure and its use in process industry can be dated back to 20 years ago [12; 13]. It still receives great research interests nowadays in nonlinear modeling and MPC applications [10; 14; 15; 16]. For dynamic modeling, lagged input and output signals should be fed into FNN to manually create dynamics[14], while RNN has naturally a dynamic structure which can be written in a nonlinear state-space form [1], thus it is very popular in process control research where dynamic modeling is desirable. As reported [1], most successful applications of RNN use the LSTM structure, which can successfully handle the gradient vanishing/exploding problem of original RNN. LSTM has already been extensively used in FDI [17], MPC [15], soft sensor [18]. AutoEncoder (AE) is another popular RNN structure whose applications are mainly in FDI and soft sensor. CNN is most well-known for its success in computer vision, it can also be used in local dynamic modeling and frequency domain modeling [18].

Great efforts have been put into research on neural networks and deep learning methods with the hope that they can promote developments of process control industry. However, unlike in the fields of CV and NLP, very few successful applications based on neural networks are reported in process industries. What can the user really benefits from a neural network based modeling, compared to linear system identification and simple nonlinear models? Most papers investigating neural networks tend to compare only the performance between several network structures and ignore this important question. Only recently, such comparison studies are carried out by several researchers. In [19], based on a 660MW boiler dataset, the model quality of LSTM is compared to those of simple (linear) statistical models and the results show that LSTM gives the worst performance. Based on the Silverbox dataset, in [14], LSTM gives worse result than FNN; in [20], the authors find that some network structures can act as noise amplifiers that deteriorate the model quality. The two papers reveal a common
phenomenon: a complex model structure mathematically capable of learning arbitrary systems well can fail in practice, even if the real system is not complex at all.

This paper further investigates the deficiencies of neural network based modeling. Besides LSTM that has been criticized in [19; 14], it will be shown that the CFNN suggested in [14; 10] also has problems that may hinder its use in modeling dynamic systems. Instead of considering modeling of nonlinear system as in [14; 10], this paper returns to the most fundamental problem: modeling of LTI systems. If a method cannot model such system well, it certainly cannot handle more complex nonlinear systems. It will be shown that, although a CFNN contains both linear part and nonlinear part, it fails to identify an LTI system without noise with consistency. Increasing the number of hidden units or hidden layers do not improve the situation. These findings will be discussed and explained through four essential aspects of system identification, model structure, error criterion, estimation properties and model validation. Simulation studies will also be presented to support the declarations.

The rest of the paper unfolds as follows. Section 2 gives background knowledge about linear system identification, LSTM and CFNN; Section 3 discusses and compares the three models from a system identification perspective, points out potential problems for neural network based modeling; Section 4 contains detailed simulation studies of a LTI system and a Hammerstein system; Section 5 gives conclusions.

### Notations

\(q\) denotes the forward time shifter. \(\sigma(\cdot)\) denotes activation function (vector) in neural networks, such as sigmoid function, Rectified Linear Unit (ReLU) function. Details about these functions can be found in [1]. \(*\) is the Hadamard (element-wise) product. \(\mathbb{N}^{+}\) denotes positive natural number. \(\Phi_{z}\) denotes power spectrum of signal \(\{z(t)\}\). Var\([\cdot]\) is the mathematical variance operator. 'With probability \(\alpha\)' is abbreviated as w.p. \(\alpha\). Euclidean norm and Frobenius norm are denoted as \(\|\cdot\|\) and \(\|\cdot\|_{F}\).

## 2 Background

Consider a general Single-Input Single-Output (SISO) LTI system [3]:

\[\mathcal{S}:y(t)=\underbrace{G_{0}(q)u(t)}_{:=y_{0}(t)}+\underbrace{H_{0}(q)e _{0}(t)}_{:=y_{0}(t)}, \tag{1}\]

where \(u(t)\) and \(y(t)\) denote system input and output signals, \(v(t)\) denotes the disturbance. \(G_{0}(q)\) denotes the system transfer function,

\[G_{0}(q)=\frac{B_{0}(q)}{A_{0}(q)}=\frac{b_{0}^{0}+\cdots+b_{0}^{n_{0}}q^{-n_{ b}}}{1+a_{0}^{1}q^{-1}+\cdots+a_{0}^{n_{c}}q^{-n_{u}}}. \tag{2}\]

which is assumed stable. For simplicity assume \(n_{a}=n_{b}=n_{e}\)\(e_{0}(t)\) is a zero-mean white noise sequence with variance \(\lambda_{0}^{2}\). \(H_{0}(q)\) is assumed to be stable and inversely stable and monic.

### The prediction error method

In system identification, a parametric model set is used to describe the true system \(\mathcal{S}\):

\[\mathcal{M}(\theta):y(t)=G(q,\theta)u(t)+H(q,\theta)e(t). \tag{3}\]

In PEM, quadratic cost function of the one-step Prediction Error (PE) is minimized:

\[\hat{\theta}_{N}=\operatorname*{arg\,min}_{\theta}\sum_{t=1}^{N} \left(y(t)-\hat{y}_{\text{pem}}(t;\theta)\right) \tag{4a}\] \[\hat{y}_{\text{pem}}(t;\theta)=\left(1-H^{-1}(q;\theta)\right)y( t)+H^{-1}(q;\theta)G(q;\theta)u(t). \tag{4b}\]

Notice that subsequently when talking about error criterion for parameter estimation, all PE refers to one-step PE. If \(\mathcal{S}\in\mathcal{M}\), which means that the model structure \(\mathcal{M}\) is flexible enough, there exists \(\theta_{0}\) such that \(G(q,\theta_{0})=G_{0}(q)\) and \(H(q,\theta_{0})=H_{0}(q)\). Moreover, if \(\mathcal{M}(\theta)\) is globally identifiable at \(\theta_{0}\) and the input signal is persistently exciting, the PEM estimate is _consistent_:

\[\hat{\theta}_{N}\rightarrow\theta_{0},\text{ w.p. 1 as }N\rightarrow\infty. \tag{5}\]

A more precise description of the conditions required for this property can be found in Chapter 8 of [3]. The consistency property implies that the estimate will approach the parameter vector representing the true system, when large amount of data are available. If \(e(t)\) in (1) is Guassian, the PEM estimate can be further proved to have _minimum variance_[3; 4].

### Long short-term memory network

The below contents show the mathematical formulation of a single-layer LSTM network. The cores of LSTM are four gates and two states controlling the transfer of the information flow. The four gates are defined of the form

\[*(t)=\sigma\left(W_{uu}u(t)+W_{ub}h(t-1)+b_{*}\right) \tag{6}\]

where \(*\) can be \(i\), \(f\), \(g\) and \(o\), corresponding to input, forget, cell and output gates respectively. Suppose that the weighting matrices and bias vectors above are of compact dimensions. Based on these gates, the cell state \(c(t)\) and hidden state \(h(t)\) are updated according to

\[c(t)=f(t)*c(t-1)+i(t)*g(t) \tag{7a}\] \[h(t)=o(t)*\tanh\left(c(t)\right). \tag{7b}\]

Finally, the output of LSTM is

\[\hat{y}_{\text{lstm}}(t)=W_{y,b}h(t)+b_{y}. \tag{8}\]

In above formulations, (6-7) constitute a LSTM layer, while (8) is often referred as a linear fully connected layer. Introduce also a vector \(\rho\) that contains all the parameters, i.e., \(W_{\text{.}}\) and \(b_{\text{.}}\) in (6-8), which can be optimized according to

\[\hat{\rho}_{N}=\operatorname*{arg\,min}_{\rho}\sum_{t=1}^{N}\left(y(t)-\hat{y} _{\text{lstm}}(t;\rho)\right)^{2}. \tag{9}\]

Notice that \(\hat{y}_{\text{lstm}}(t)\) is calculated according to (6-8). To obtain a deep network structure, one can simply connect several LSTM layer in series.
### Cascade forward neural network

CFNN here refers to a specific FNN suggested in [14] for nonlinear identification. It is characterized by that outputs of all previous units are used as inputs in the next layer. It is already available in Matlab's System Identification Toolbox, as an idnlarx object. As has been mentioned in Section 1, lagged input and output signals should be used to introduce dynamics. The input of CFNN is the regressor

\[\varphi^{\top}(t)=\left[y(t-1),\cdots,y(t-n),u(t),\cdots,u(t-n)\right]. \tag{10}\]

The output of a single layer CFNN is

\[\hat{y}_{\text{cfnn}}(t)=\underbrace{W_{yu}\varphi(t)+b_{y}}_{\text{Linear}}+ \underbrace{W_{yu}\sigma\left(W_{hu}\varphi(t)+b_{h}\right)}_{\text{Nonlinear}}. \tag{11}\]

Collect all parameters of CFNN into a vector \(\vartheta\), it can be optimized according to

\[\hat{\vartheta}_{N}=\operatorname*{arg\,min}_{\vartheta}\sum_{t=1}^{N}\left(y (t)-\hat{y}_{\text{cfnn}}(t;\vartheta)\right)^{2}. \tag{12}\]

As shown in (11), CFNN can be divided into linear and nonlinear parts where the nonlinear part is due to the hidden layer. Similar to LSTM, one can use multiple hidden layers to achieve a deep structure.

## 3 Neural network based dynamic modeling from an identification perspective

Shown in Fig. 1 is a typical identification procedure [3; 4; 21] covering essential steps and key points of a general black-box dynamic modeling. This section discusses the three models introduced in Section 2 then points out problems that are often neglected. The experiment design and parameter optimization problems will not be discussed, only the steps with black dotted line in Fig. 1 are focused.

### Model structure

Among the three methods listed in Section 2, the structure of PEM is exactly the same as the true system \(\mathcal{S}\). By introducing a new state vector \(x^{\top}(t)=[c^{\top}(t),h^{\top}(t)]\), LSTM model (6-8) can be summarized to a NonLinear State-Space (NLSS) form [1]:

\[x(t)=\mathcal{F}\left(x(t-1),u(t);\rho\right) \tag{13a}\] \[\hat{y}_{\text{ksm}}(t;\rho)=\mathcal{H}\left(x(t);\rho\right). \tag{13b}\]

Notice that it is assumed that no time delay exists in input and output in (13). According to [14; 10], CFNN belongs to the Nonlinear ARX (NARX) model:

\[\hat{y}_{\text{cfnn}}(t;\vartheta)=\mathcal{G}\left(\varphi(t);\vartheta \right). \tag{14}\]

In (13-14) \(\mathcal{F}\), \(\mathcal{H}\), \(\mathcal{G}\) are user-defined (nonlinear or linear) functions.

As has been mentioned in Section 2, CFNN model contains a linear part. When \(W_{yu}\), \(W_{hu}\), \(b_{y}\) and \(b_{h}\) all vanish and if

\[W_{yu}=[-a_{0}^{1},\cdots,-a_{0}^{n_{h}},b_{0}^{1},\cdots,b_{0}^{n_{h}}], \tag{15}\]

CFNN will present exactly the same behavior as \(G_{0}(q)\). That is to say, CFNN covers the LTI model structure as its special case and is theoretically capable of modeling such system, which resembles Volterra model [22], block-oriented model [23], linear parameter-varying model [24], etc. In contrast, it is non-trivial how LSTM network can be connected or reduced to a common LTI model. In [19], after a careful design of the hyperparamters LSTM still gives worst performance among other statistical models. The authors attribute this phenomenon to the effects of gate functions in (6), disabling some of these gates can improve the performance of LSTM. Simulation results in [14] are similar.

The above discussions and the results in [14; 19] reveal that, for process dynamic modeling, models that cannot be interpreted from process dynamics are undesirable even if they have complex structures with strong approximation abilities.

### Error criteria

Error criteria are the basics for optimization of model parameters, they are application-oriented and are closely related to the properties of the estimated model. Three commonly used error criteria of the linear model set (\(\mathcal{M}\)) are, Prediction Error (PE), Output Error (OE, also called simulation error), Equation Error (EE):

\[\varepsilon_{\text{pe}}(t;\theta) =H^{-1}(q;\theta)\left(y(t)-G(q;\theta)u(t)\right) \tag{16a}\] \[\varepsilon_{\text{oe}}(t;\theta) =y(t)-G(q;\theta)u(t)\] (16b) \[\varepsilon_{\text{oe}}(t;\theta) =A(q;\theta)y(t)-B(q;\theta)u(t) \tag{16c}\]

where \(A(q)\) and \(B(q)\) are defined similarly to \(A_{0}(q)\) and \(B_{0}(q)\) in (2). Notice that EE equals PE of an ARX model [3] hence only PE and OE will be considered. The difference between the predicted output (4b) and the simulated output \(G(q;\theta)u(t)\) is that predicted output uses measured output signals while simulated

Figure 1: Typical identification procedure.
output only uses input signals. Bearing this point in mind, the extensions in LSTM and CFNN are straightforward: because no measured outputs are used in \(\hat{y}_{\text{lstm}}\), the LSTM criterion (9) is an output error criterion; the CFNN criterion (12) is a prediction error (equation error) criterion.

While the PEM model set \(\mathcal{M}\) has a noise model that can be parameterized independently to process model (e.g., Box-Jenkins (BJ) model [3]), LSTM and CFNN are not equipped with such ability to handle disturbance. In [20], the authors argue that the noise effects can even be amplified in CFNN criterion (12) and suggests the use of output error criterion. However, how to optimize CFNN model under an output error criterion is an unsolved problem.

**Remark 1**: _Sometimes the input of LSTM is chosen to be a regressor containing measured outputs like \(\varphi(t)\) in (10), which is called teacher forcing in Chapter 10.2.1 in [1]. In this case, the LSTM criterion (9) becomes a prediction error criterion._

### Estimation properties

Analyzing the (statistical) properties of the estimated model is of particular interest and of vital importance in all modeling technique. The total model error can be divided into structure error caused by deficiencies in model structure and random model error caused by stochastic disturbance. For linear identification, one often investigates the consistencies and variances of parameters, step responses or frequency responses. The two points concerning neural network will be respectively discussed below.

#### Consistency

For nonlinear model structures, it makes less sense to consider in the parameter space. Instead, the _consistency in step response_ can be defined analogously to (5):

**Definition 1**: _Denote \(\{\mathcal{U}(t)\}\) as a step signal of amplitude \(A_{\mathcal{U}}\). Then for some general system \(\mathcal{S}\) and an estimated model \(\hat{\mathcal{M}}_{N}\) under structure \(\mathcal{M}\) and with \(N\) data samples, as well as their outputs subject to \(\{\mathcal{U}(t)\}\): \(\{\mathcal{Y}_{0}(t)\}\) and \(\{\hat{\mathcal{Y}}_{N}(t)\}\), if_

\[\hat{\mathcal{Y}}_{N}(t)\rightarrow\mathcal{Y}_{0}(t),\text{ w.p. 1 as }N\rightarrow\infty,\ \forall t\in\mathbb{N}^{+},A_{\mathcal{U}}\neq 0, \tag{17}\]

_the estimate of \(\mathcal{M}\) is consistent in step response to \(\mathcal{S}\)._

In this regard, the estimated model \(\hat{\mathcal{M}}_{N}\) can accurately describe the input-output relations of the true system \(\mathcal{S}\) if sufficient data samples are collected. For LTI system, PEM can deliver consistent estimate in parameters, step response and frequency response (under some conditions, see Section 2.1), and for step response one only needs to consider the case \(A_{\mathcal{U}}=1\).

It is necessary to give some discussions on the differences between the above consistency concept and the well-known _universal approximation theorem_[1; 5; 6]. In Chapter 6.4.1 of [1], this theorem is summarized as:

_An FNN with a linear output layer and at least one hidden layer with the commonly used activation function (sigmoid, ReLU, etc.) can approximate any Borel measurable function from one-dimensional space to another with any desired amount of error._

The theorem states that even a single-layer FNN (which is nearly the simplest network) is theoretically capable of approximating any functions arbitrarily well, which seems to imply that neural network is a powerful modeling tool. But note that this is only an existence theorem that gives neither the guarantee of consistent estimates, nor the guideline for how to obtain a consistent estimate using training data. This theorem may have made control community overly optimistic about the modeling capability of neural networks for dynamic systems.

#### Variance

Consider first the asymptotic variance expression of the frequency response for linear system [25]:

\[\text{Var}\left[\hat{G}_{N}^{n}(\text{e}^{\text{i}\omega})\right]\approx\frac {n}{N}\frac{\Phi_{v}(\omega)\lambda_{0}}{\Phi_{u}(\omega)\lambda_{0}-|\Phi_{u \omega_{0}}(\omega)|} \tag{18}\]

where \(\hat{G}_{N}^{n}(\text{e}^{\text{i}\omega})\) denotes frequency response function of a \(n\)th-order estimate based on \(N\) data samples. The expression holds exactly for ARX model when \(N\rightarrow\infty\), \(n\rightarrow\infty\). It reveals that the model variance is proportional to number of parameters and is inversely proportional to sample size. Although there is no such conclusion available for nonlinear systems and neural network models, it is generally acknowledged that complex models containing many parameters are more vulnerable to stochastic disturbances.

Consider modeling a \(n\)th order SISO LTI system like \(\mathcal{S}\), suppose that the correct order is used for PEM (a BJ model is used) and CFNN, all biases in LSTM and CFNN are set to zero, and use \(n_{h}\) hidden units for both LSTM and CFNN. Then the parameter numbers of the three models are

BJ: \[n_{\theta} =4n\] (19a) LSTM: \[n_{\rho} =4\left(n_{h}^{2}+n_{h}\right)+n_{h}\] (19b) CFNN: \[n_{\theta} =n_{h}\left(2n+1\right)+2n\] (19c)

Notice that only single-layer LSTM and CFNN with fully connected output layers are considered.

Fig. 2 shows the parameter numbers of different models with varying system orders and hidden unit settings. Among three model structures, LSTM has the most parameters and is 1-4 orders of magnitude higher than BJ model; CFNN ranges in the

Figure 2: Parameter numbers of different models under different system orders.
middle. For dynamic process modeling, the training data must be informative enough to contain the main process behavior, which can be guaranteed by adding persistently exciting test signals to the plant [21]. This procedure certainly introduces some slight disturbance to normal operations and is the main cost of black-box process modeling. Heuristically, according to (18), to achieve a same level variance, LSTM requires 1-4 orders of magnitude more data than BJ model, which means a huge increase in modeling cost. This gives another explanation about the poor model quality in [14; 19] apart from the model structure issue.

### Model validation

Model validation is the final step before a model can be put into use. It should be application-oriented. In MPC, the model is used to give multi-step predictions. If the step size of a multi-step prediction tends to infinity, it becomes simulated output, see Chapter 3.2 of [3] for details. Recently, it is proved that output error serves as a useful tool for FDI as well [26]. Hence it is important to check simulation error because it reflects the real gap between the model and the plant. However in many recently published papers, the model quality is only validated through (one-step) prediction error. In [27], the author illustrated that two model with close prediction errors can have huge differences in simulation errors.

Typically, a neural network model with complex structure can easily deliver a loss function of very small value. When the loss corresponds to prediction error and the model gives accurate one-step prediction, one can tell nothing about the model quality from the input to the output. If possible, check simulation error; or at least test the multi-step prediction error for validation.

### Summary

LSTM has a very different structure from a LTI system hence it may have difficulties to model such a system. In contrast, CFNN has a structure which can be interpreted as a linear model extension. However, as discussed above, there is no guarantee for its model consistency and it may have high variance. Further, a small prediction error delivered by CFNN may not imply a good input-output model.

## 4 Simulation study

While LSTM has been comprehensively tested in [14; 19], this section studies CFNN in a linear ARX system and a Hammerstein system. Step responses under different input amplitudes will be calculated and normalized according to the input amplitudes for comparison. Mean values are removed in the training data so the bias vectors in CFNN will be disabled.

Simulations performed in this section are all performed using Matlab and System Identification Toolbox. Specifically, nlarx is used to estimate CFNN; BJ is used to estimate a BJ model; nlhw is used to estimate a Hammerstein system. For above estimates obtained in System Identification Toolbox, sim and predict are used to calculate model simulation and \(k\)-step prediction.

### Linear system

Consider the following ARX system:

\[\begin{split} y(t)=\frac{B_{0}(q)}{A_{0}(q)}u(t)+\alpha\frac{1}{ A_{0}(q)}e_{0}(t)\\ B_{0}(q)=0.0115+0.00639q^{-1}\\ A_{0}(q)=1-1.963q^{-1}+0.965q^{-2}\end{split} \tag{20}\]

where \(\alpha\) is used to control the Noise-to-Signal Ratio (NSR), defined as \(\mathrm{Var}\left[v_{0}^{2}(t)\right]/\mathrm{Var}\left[v_{0}^{2}(t)\right]\), see (1) for definitions of \(y_{0}(t)\) and \(v_{0}(t)\).

#### 4.1.1 Noise-free system

The system without noise is considered. The training data are generated with a Generalized Binary Noise (GBN) [28] as input. Its average switching time is 20 samples and amplitude is 1. The sample size \(N=50000\), the result below does not change if \(N\) is further increased. Subsequently, a GBN signal with amplitude \(A\) and average switching time \(T\) will be abbreviated as \(A*\mathrm{GBN}(T)\) for simplicity. The input in the validation data have 5 different types: \(0.2*\mathrm{GBN}(20)\), \(1*\mathrm{GBN}(10)\), \(1*\mathrm{GBN}(20)\), \(1*\mathrm{GBN}(40)\), \(5*\mathrm{GBN}(20)\). Each input lasts for 1000 samples. Among these inputs, the first and the fifth have different amplitudes compared to the one in training data but have the same spectral distribution; the second and the fourth have different spectral distributions but have the same amplitude; the third is entirely the same. A CFNN with one layer and 4 hidden units is trained. All activation functions are ReLUs, which are suggested to use in [10; 14]. The model validation based on 1-step prediction error and output error are shown in

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Residual & 0.2*GBN(20) & 1*GNN(10) & 1*GNN(20) & 1*GNN(40) & 5*GNN(20) \\ \hline
1-step PE & 9.99E-07 & 8.38E-08 & 2.80E-08 & 1.61E-08 & 1.14E-09 \\
20-step OE & 1.81E-02 & 1.12E-03 & 1.41E-04 & 3.05E-04 & 2.17E-04 \\ Simulation & 5.73E+00 & 1.20E-02 & 3.56E-03 & 3.56E-03 & 5.12E-03 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Noise-free LTI system: REs(%) of residuals in different data sections.

Figure 3: Model validation based on one-step prediction and simulation.
to the true one are recorded in the second raw of Table 2. The FIT between two sequence \(Z_{0}\) and \(\hat{Z}\) is

\[\text{FIT}=1-\frac{\left\|Z_{0}-\hat{Z}\right\|}{\left\|Z_{0}-\text{mean}(\hat{Z })\right\|}. \tag{21}\]

The best result is achieved when input amplitude equals 1. This is because GBN is a binary signal that only have two values. For amplitudes different from the one in training data, the step responses of CFNN deviate from the true one.

Three other types of inputs are also tested: 2*GBN(20), zero-mean Gaussian white input with variance \(0.33^{2}\), uniform input that ranges in \((-1,1)\). The results are shown in Fig. 4(c-d) and Table 2. It is interesting to note that for 2*GBN(20) the best FIT moves to amplitude 2; for Gaussian input the best FIT occurs for the smallest amplitude 0.2 because Gaussian distribution has a bell-shaped curve; for uniform input the gaps between the best and the other are insignificant because the curve of uniform distribution is flat in its range.

For each case, the MSE of training data have been optimized to very small value (\(<10^{-6}\)) and obtained models differ, which implies that CFNN that can give PE nearly to zero is non-unique. The final estimated CFNN is strongly dependent on the input characteristics, the initial conditions, etc. When encountering a 'never-met' (does not occur in training data) or 'unfamiliar' (does not occur frequently in training data) input, CFNN gives wrong step response.

\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline Input type & 0.2 & 0.5 & 1 & 2 & 5 \\ \hline \(1^{+}\)GBN(20) & 50.59 & 85.95 & 99.79 & 93.23 & 89.09 \\ \(2^{*}\)GBN(20) & 19.67 & 60.94 & 85.95 & 99.79 & 91.85 \\ Gaussian & 83.81 & 75.12 & 72.26 & 70.85 & 70.01 \\ Uniform & 80.31 & 94.96 & 87.66 & 83.87 & 81.60 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Noise-free LTI system: FITs(%) of different step responses. The numbers in the first raw are input amplitudes. The red one denotes the best for each input.

Figure 4: Noise-free LTI system: step responses under different inputs.

Figure 5: Noisy LTI system: comparison of step responses of BJ and different CFNN structures when NSR=5%. For CFNN, the best step response among input amplitude \([0.2,0.5,1,2,5]\) in each run is plotted. The edge of the red region is the envelope of 100 runs.

\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline \multirow{2}{*}{
\begin{tabular}{l} NSR (\%) \\ \end{tabular} } & \multicolumn{5}{c}{CFNN (with different input amplitudes)} \\ \cline{2-7}  & 0.2 & 0.5 & 1 & 2 & 5 & Best \\ \hline
0 & 100.00 & 63.43 & 89.35 & 99.23 & 94.42 & 91.22 & 99.26 \\
5 & 95.67 & 40.84 & 77.30 & 90.48 & 85.16 & 80.21 & 93.50 \\
10 & 93.64 & 39.54 & 73.85 & 88.67 & 83.48 & 78.21 & 92.50 \\
20 & 92.42 & 28.68 & 66.80 & 83.30 & 76.84 & 70.22 & 89.08 \\
40 & 86.61 & 27.35 & 62.78 & 76.50 & 68.80 & 61.88 & 84.62 \\ \hline \end{tabular}
\end{table}
Table 3: Noisy LTI system: comparison of BJ and CFNN in mean FITs (%) of step responses. CFNN has one layer with 4 hidden units.

Figure 3: The Relative Errors (RE) of one-step prediction, simulation and additionally 20-step prediction are shown in Table 1. RE is calculated according to \(\text{Var}\left[\varepsilon^{2}(t)\right]/\text{Var}\left[y^{2}(t)\right]\) where \(\varepsilon\) is some residual and \(y\) the measured output.
#### 4.1.2 Noisy system

In this part, the performance of CFNN will be compared to PEM for noisy system. The true system has an ARX structure, the parameter optimization of the same model has a closed-form solution. However, a BJ model that can also give consistent estimate is used. In this case, both CFNN and BJ require numerical optimization. For each NSR setting, 100 Monte Carlo simulations are run. In each simulation, 10000 training data are generated under the input is 1*GBN(20), then the step response of CFNN is calculated under different input amplitudes. Among these responses the one that gives the best FIT will be recorded.

A single-layer CFNN with 4 hidden units is first tested, the results are shown in Table 3. CFNN delivers worse results than BJ for all NSRs, even for the best ones chosen from the five candidates. Consistent with the discussions in Section 4.1.1, the mean FITs under input amplitude 1 give best result. Additionally, Table 4 presents information about the mean estimated parameters of CFNN. Recall that \(W_{hu}\) and \(W_{yh}\) corresponds to the nonlinear part and \(W_{wu}\) corresponds to the linear part. Concerning the linear part, \(W_{yu}(1:2)\) related to \(A_{0}(q)\) is consistent while \(W_{yu}(3:4)\) related to \(B_{0}(q)\) is not, \(\|W_{yu}(3:4)\|\) decreases as NSR decreases; in the nonlinear part, \(\|W_{uk}\|_{F}\) and \(\|W_{yh}\|\) increases as NSR increases. This reveals that there is a competition between the weighting matrices of nonlinear and linear parts. When noise level increase, the nonlinear part takes the advantage gradually and make the results deviate further away from the true system. Notice that the consistency of the parameters related to \(A_{0}(q)\) does not hold generally. When other types of inputs are used, such as Gaussian, uniform, all parameters become inconsistent.

Different structure settings of CFNN are tested and the mean FITs are shown in Table 5. One can see that increasing numbers of hidden units or layers do not improve the situation. In fact, the poorest result is obtained when three layers are used. The step responses of 100 Monte Carlo runs of three selected cases are plotted in Fig. 5. The more complex the structure of CFNN, the higher the variance; the best setting also has higher variance than BJ.

#### 4.1.3 Summary and discussion

The simulation results give surprise finding: as a universal approximator, CFNN cannot even give consistent estimate for a simple LTI system that is contained in its model structure. Although when all weighting matrices in its nonlinear part vanish, CFNN is simply a linear ARX model, there is no guarantee that estimated model is consistent. The 'flexible' structure of CFNN, enabled by the nonlinear part, i.e., the hidden layers, becomes a nuisance factor for identification of LTI system. For a nonlinear system that is more complex than a LTI system, such neural network based models can perform poorer.

### Hammerstein system

Consider the following Hammerstein system:

\[\begin{split}& y(t)=\frac{B_{0}(q)}{A_{0}(q)}f\left(u(t) \right)+\alpha\frac{1}{A_{0}(q)}e_{0}(t)\\ & f(u)=10u^{3}+3.5u^{2}+u.\end{split} \tag{22}\]

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline Model & -2.5 & -1.5 & -0.5 & 0.5 & 1.5 & 2.5 \\ \hline Polynomial Hammerstein & 94.61 & 94.41 & 52.81 & 77.83 & 96.12 & 95.55 \\ CFNN (\(m=3,m=3\)) & -1.13 & 88.79 & 27.42 & 45.03 & -3.52 & 14.50 \\ CFNN (\(m=4,m=4\)) & -0.39 & 87.01 & 27.13 & 56.05 & -3.54 & 32.76 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Noisy Hammerstein system: mean FITs(%) of step responses. The numbers in the first raw are input amplitudes.

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline NSR (\%) & \(\|W_{m}\|_{F}\) & \(\|W_{yu}\|\) & \(\|W_{yu}(3:4)\|\) & \(W_{yu}(1)\) & \(W_{yu}(2)\) & \(W_{yu}(3)\) & \(W_{yu}(4)\) \\ \hline
0 & 2.43 & 7.33E-05 & 9.48E-04 & 1.9630 & -0.9650 & 8.28E-04 & 4.62E-04 \\
5 & 2.45 & 3.60E-04 & 8.77E-04 & 1.9630 & -0.9650 & 7.62E-04 & 4.33E-04 \\
10 & 2.46 & 6.16E-04 & 8.02E-04 & 1.9627 & -0.9648 & 6.94E-04 & 4.04E-04 \\
20 & 2.49 & 8.92E-04 & 7.59E-04 & 1.9631 & -0.9650 & 6.56E-04 & 3.81E-04 \\
40 & 2.57 & 1.51E-03 & 7.32E-04 & 1.9627 & -0.9647 & 6.99E-04 & 2.20E-04 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Noisy system: information about the mean estimated parameters of CFNN.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \(n_{l}\) & \(n_{b}\) & 0.2 & 0.5 & 1 & 2 & 5 & Best \\ \hline
1 & 1 & 41.80 & 76.04 & 90.26 & 85.40 & 80.68 & 93.81 \\
1 & 2 & 47.35 & 77.84 & 88.85 & 84.24 & 79.35 & 92.31 \\
1 & 4 & 40.84 & 77.30 & 90.48 & 85.16 & 80.21 & 93.50 \\
1 & 6 & 50.04 & 78.37 & 89.31 & 86.95 & 83.36 & 92.80 \\
1 & 10 & 40.84 & 70.97 & 81.03 & 78.90 & 75.68 & 83.47 \\
1 & 4 & 41.80 & 76.04 & 90.26 & 85.40 & 80.68 & 93.81 \\
2 & 4 & 8.31 & 21.12 & 81.98 & -3.09 & -36.67 & 82.24 \\
3 & 4 & -12.34 & -4.88 & 43.76 & -223.69 & -282.30 & 63.32 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Noisy LTI system: mean FITs of different CFNN structures. \(n_{l}\) denotes the layer number, \(n_{h}\) denotes hidden units in each layer. The red one denotes the best settings.

Figure 6: Noise-free Hammerstein system: comparison of step responses under different input amplitudes. CFNN with 3 layers and 3 hidden units in each layer is used.
The linear part of this system is chosen entirely the same as (20). 10000 training data are generated using a Generalized Multiple-level Noise (GMN) signal (see Chapter 9.1.2 of [21]) with average switching time 10s and amplitude ranging in \([-2,2]\). When using idnlhw function the orders of system and polynomial are set to their correct values. Slightly different from the LTI case, in the first layers the activation functions are ReLU while in other layers they are hyperbolic tangent functions.

Fig. 6 shows the result of noise-free case. The step responses delivered by the polynomial Hammerstein model coincides with the true system. CFNN only gives consistent results for amplitude -1.5 and 1.5; for other amplitudes, the results are very poor. The noisy system with 1% is also tested, shown in Table 6. The polynomial Hammerstein has better FITs than CFNN for all cases tested. The variances of step responses are similar to Fig. 5 in which CFNN delivers large variances, and it will not be shown here for brevity.

## 5 Concluding remarks

Many researchers in control community are optimistic about the use of neural networks for dynamic system modeling, perhaps due to their success in CV and NLP. In this work, three representative models PEM, CFNN and LSTM are compared for their ability in LTI system identification. As reported, LSTM is unsuitable for dynamic system identification. CFNN has a reasonable structure and can be reduced to a common LTI model. However, no results exist to guarantee the model consistency. Moreover, the large number of model parameters of LSTM and CFNN will result in large model variance. In simulation studies of the LTI system, CFNN fails to give consistent step responses even in the noise-free case. In the noisy case, CFNN models have larger model variances than the BJ model. When tested in a Hammerstein system, CFNN gives poorer performance. Increasing hidden unit number and hidden layer number do not improve model quality.

This study reveals that there is still a long way to go for neural network based dynamic system identification/modeling. Following remarks can be made based on the findings:

1. The success of neural network models in CV and NLP does not guarantee its success in dynamic system modeling and control;
2. in the noise-free case, in numerical optimization for parameter estimation a neural network model may not converge to a dynamic model that is contained in its model structure when the loss function tends to zero;
3. the theorem of universal approximator cannot guarantee model consistency;
4. if a neural network model is unsuitable for modeling an LTI system, it will have more difficulties to model a nonlinear dynamic system;
5. the performance of neural network based dynamic system modeling should be compared to that of traditional linear and simple nonlinear system identification.
