2022

[1]Jorge S. S. Júnior

[1]Department of Electrical and Computer Engineering, Pólo II, University of
Coimbra, Institute of Systems and Robotics, Coimbra, 3030-290, Portugal
2]Dept. Analytical Chemistry & Chemometrics, Radboud University, Nijmegen,
6525 AJ, the Netherlands

# Survey on Deep Fuzzy Systems in regression applications: a view on
interpretability

<EMAIL_ADDRESS>Jérôme Mendes<EMAIL_ADDRESS>Francisco Souza
<EMAIL_ADDRESS>Cristiano Premebida<EMAIL_ADDRESS>*  [

###### Abstract

Regression problems have been more and more embraced by deep learning (DL)
techniques. The increasing number of papers recently published in this domain,
including surveys and reviews, shows that deep regression has captured the
attention of the community due to efficiency and good accuracy in systems with
high-dimensional data. However, many DL methodologies have complex structures
that are not readily transparent to human users. Accessing the
interpretability of these models is an essential factor for addressing
problems in sensitive areas such as cyber-security systems, medical, financial
surveillance, and industrial processes. Fuzzy logic systems (FLS) are
inherently interpretable models, well known in the literature, capable of
using nonlinear representations for complex systems through linguistic terms
with membership degrees mimicking human thought. Within an atmosphere of
explainable artificial intelligence, it is necessary to consider a trade-off
between accuracy and interpretability for developing intelligent models. This
paper aims to investigate the state-of-the-art on existing methodologies that
combine DL and FLS, namely deep fuzzy systems, to address regression problems,
configuring a topic that is currently not sufficiently explored in the
literature and thus deserves a comprehensive survey.

###### keywords:

Deep Regression, Explainable Artificial Intelligence, Interpretability, Deep
Fuzzy Systems

##  1 Introduction

The goal in regression is to predict one or more variables
$\bm{\mathbf{y}}(k)=(y_{1}(k),\ldots,y_{m}(k))^{T}$ from the information
provided by measurements $\bm{\mathbf{x}}(k)=(x_{1}(k),\ldots,x_{p}(k))^{T}$
for a given sample $k$. Customary, $\bm{\mathbf{y}}(k)$ are refereed as
targets, outputs, or dependent variables, while $\bm{\mathbf{x}}(k)$ are
commonly refereed as predictors, inputs, covariates, regressors, or
independent variables. Regression models covers several application areas, as
economic growth problems Busu2019 ; Botev2019 , air quality prediction
Liu2021b ; Gu2018b , medicine Zhu2020 ; Shi2019b , chemical industries
Orlandi2021 ; Yang2021 , and industrial processes Souza2021 ; Liu2019 . Recent
studies show that regression models have become predominant in increasingly
complex real-world systems due to the large availability of data, inclusion of
nonlinear parameters, and other aspects intrinsic to the application area. For
complex systems, traditional machine learning techniques (i.e., non-
deep/shallow techniques) may become limited as they face two main
characteristics in current systems: high-dimensionality (with a high amount of
observations and inputs) and complexity (due to the variety of dynamic and
nonlinear features, hyperparameters, and uncertainty). Deep learning (DL)
techniques have gained prominence in recent years due to their ability to
represent systems in complex structures with multiple levels of abstraction
and high-level features. However, deep learning techniques may have
limitations such as the need for a sufficiently large dataset for model
training, sensitivity to hyperparameter selection, and interpretability issues
Sun2021 ; Angelov2020 . In this sense, deep fuzzy systems have emerged as a
viable methodology to balance accuracy and interpretability in complex real-
world systems.

Regression models can be categorized as Sjoberg1995 : (i) “white-box” when the
input-output mapping is built upon first-principle equations, (ii) “black-box”
when the mapping is derived from the data (also referred to as data-driven
modeling), or (iii) “grey-box” when the knowledge about the input-output
mapping is known beforehand and integrated along with data-driven modeling.
White-box models are advantageous for promoting interpretations of the
internal mechanisms associated with input-output mapping. On the other hand,
black-box models can address complex systems with predictive analysis without
prior knowledge of the system. Grey-box models can combine the
interpretability presented by “white-box” and the ability to learn from data
given by “black-box” models (e.g., fuzzy systems). Regarding data-driven
modeling, the dependency between input and output can be built by linear or
nonlinear models. A regression model is linear when the rate of change between
input-output is constant due to the linear combination of the inputs. Examples
include the multivariate linear regression models. Data-driven nonlinear
regression is adopted when the input-output dependence is nonlinear and can
not be covered by linear modeling. There is a plethora of methods for
nonlinear regression, and its applicability is problem-dependent. Examples
include fuzzy systems, support vector regression, artificial neural networks
(e.g., non-deep/shallow and deep networks), and rule-based regression (e.g.,
decision trees and random forest).

Recent surveys and reviews have showcased the application of DL to regression
problems. The work of Han et al. Han2021  reviews deep models for time-series
forecasting, where the models are categorized as: (i) discriminative, where
the learning stage is based on the conditional probability of the
output/target given an observation, (ii) generative, which learn the joint
probability of both output and observation, with the generation of random
instances), or (iii) hybrid, a combination of different deep methods. With the
implementation in benchmark systems and a real-world use case related to the
steel industry, the authors showed that deep models are efficient in
discriminating complex patterns in time series with high-dimensional data. Sun
et al. Sun2021  discuss the use of DL for soft sensor applications, showing
the trends and applications in industrial processes, in addition to best
practices for model development. The authors established some directions for
future research, such as solutions to address the lack of labeled samples
(e.g., semi-supervised methods), hyperparameter optimization, solutions to
improve model reliability (e.g., model visualization), and the development of
DL methods with distributed and parallel modeling. Torres et al. Torres2021
explore deep learning for time-series forecasting, together with time-series
definition and processing using deep architectures commonly described in the
literature. In addition, the authors demonstrated some practical aspects of
using DL methods to solve complex problems with big data, such as a variety of
libraries and techniques compatible with deep structures for automatic
optimization of hyperparameters, hardware infrastructure to benefit DL
implementation with optimizable complexity and processing time, and
flexibility to address real-world applications. Pang et al. Pang2021  and
Chalapathy et al. Chalapathy2019  present an overview of DL studies for
anomaly detection; they also discuss the complexities and different types of
models (e.g., classification, autoregressive, unsupervised, and semi-
supervised) applied in various intelligent systems, such as cyber-security
systems, medical monitoring, financial surveillance, and industrial processes.
Other works that review the DL literature on regression are Dong2021 ;
Sengupta2020 ; Pouyanfar2018 . It is noted from these works that DL techniques
have some advantages over non-deep methods, such as the ability to learn
complex representations with automatic feature engineering, not requiring
prior experience or knowledge, good performance by increasing the
dimensionality of the data, among other specific advantages depending on the
type of framework and application Navamani2019 .

Despite the advantages portrayed in the literature, deep learning has
limitations, such as the need for a sufficiently large dataset for model
training, sensitivity to hyperparameter selection, and lack of
interpretability Sun2021 ; Angelov2020 . Also, there is a lack of proper
explanations of the internal structure of deep model structures, which raises
concern in applications that directly and indirectly impact human life as well
for operational decisions Angelov2021b ; Confalonieri2021 . Having this
concern in mind, recent works address the interpretability issue of DL from
the following principles of “eXplainable Artificial Intelligence” (XAI)
systems Phillips2021 : the existence of an appropriate explanation for each
decision made; each explanation must be meaningful to the user; the process
must accurately consider what happens in the system; identify the situations
in which the system may or may not function properly (knowledge limits).

Fuzzy logic systems (FLS), composed of IF-THEN rules with linguistic terms
mimicking human thought Chimatapu2018 , is one of the research areas that
contemplates the XAI principles. FLS has a wide range of approaches to
nonlinear systems, primarily in terms of interpretability, whether related to
the complexity and semantics of fuzzy rules, notation readability, coverage of
input data space (operation regions), and so on Lapa2017 ; as2020 . Moral et
al. Moral2021  show the main benefits of adopting FLS for the development of
explainable methodologies, with a discussion of fundamental concepts and
definitions associated with XAI and FLS to how to design increasingly
interpretable models. Therefore, DL techniques can be complemented with FLS,
developing deep fuzzy systems (DFS) and then providing an easy-to-understand
and easy-to-implement interface to efficiently address the main drawbacks of
DL and FLS, thus ensuring good accuracy and good interpretability. The surveys
of Das2020  and Zheng2021  investigated some recent trends in DFS models and
real-world applications (e.g., time series forecasting, natural language
processing, traffic control, and automatic control). However, despite the
benefit of adopting FLS principles to DL systems, no comprehensive survey or
review has been conducted focusing exclusively on deep fuzzy for regression
problems.

This paper surveys and discusses the state-of-the-art on deep fuzzy techniques
developed to deal with a diverse range of regression applications. Initially,
Section 2 presents fundamental concepts about FLS and XAI. Then, an overview
of DL techniques commonly used for regression will be presented in Section 3,
namely Convolutional Neural Networks, Deep Belief Networks, Multilayer
Autoencoders, and Recurrent Neural Networks. Next, Section 4 shows the
literature on deep fuzzy systems in two ways: (i) standard deep fuzzy systems,
based on fundamental FLS; (ii) hybrid deep fuzzy systems, with the combination
of FLS and the conventional deep models discussed in Section 3. Finally,
Section 5 presents general discussions based on the state-of-the-art surveyed.

##  2 Background

###  2.1 Fuzzy logic systems

Developed initially by Lotfi A. Zadeh Zadeh1965 , FLS are rule-based systems
composed mainly of an antecedent part, characterized by an “IF” statement, and
a consequent part, characterized by a “THEN” statement, allowing the
transformation of a human knowledge base into mathematical formulations, thus
introducing the concept of linguistic terms related to the membership degree.

The basic configuration of a fuzzy logic system, shown in Figure 1,

Figure 1: Basic configuration of fuzzy logic systems.

depends on an interface that transforms the real input variables into fuzzy
sets (fuzzifier), which are interpreted by a fuzzy inference model to perform
an input-output mapping based on fuzzy rules. Thus, the mapped fuzzy outputs
go through an interface that transforms them into real output variables
(defuzzifier) Wang1997 ; ci2011 . Some well-known fuzzy systems are Mamdani
fuzzy systems Mamdani1975 , Takagi-Sugeno (T-S) fuzzy systems Takagi1985 , and
Angelov-Yager’s (AnYa) fuzzy rule-based systems using data clouds Angelov2011
. Among these, the T-S fuzzy models stand out for their ability to decompose
nonlinear systems into a set of linear local models smoothly connected by
fuzzy membership functions Qiu2016 . T-S fuzzy models are universal
approximators capable of approximating any continuous nonlinear system, that
can be described by the following fuzzy rules Ying1997 :

$\displaystyle
R^{i}:\textbf{IF}~{}x_{1}(k)~{}\text{is}~{}F_{1}^{i}~{}\text{and}~{}\ldots~{}\text{and}~{}x_{p}(k)~{}\text{is}~{}F_{p}^{i}$
$\displaystyle\textbf{THEN}~{}\bm{\mathbf{y}}^{i}(k)=\bm{\mathbf{f}}^{\,i}(x_{1}(k),\ldots,x_{p}(k)),$
(1)

where $R^{i}$ ($i=1,\ldots,N$) represents the $i$-th fuzzy rule, $N$ is the
number of rules, $x_{1}(k),\ldots,x_{p}(k)$ are the input variables of the T-S
fuzzy system, $F^{i}_{j}$ are the linguistic terms characterized by fuzzy
membership functions $\mu^{i}_{F^{i}_{j}}$, and
$\bm{\mathbf{f}}(x_{1}(k),\ldots,x_{p}(k))$ represents the function model of
the system of the $i$-th fuzzy rule Junior2021 .

###  2.2 Explainable artificial intelligence

Although the explainable artificial intelligence (XAI) concept is often
associated with a homonymous program formulated by a group of researchers from
the Defense Advanced Research Projects Agency (DARPA) Hall2019 , the
principles related to explainability gained strength from the 1970s onwards.
The earliest works presented rule-based structures and decision trees with
human-oriented explanations, such as the MYCIN system proposed in
Shortliffe1973  developed for infectious disease therapy consultation, the
tutoring program GUIDON proposed in Clancey1979  based on natural language
studies, among numerous other systems Weiss1978 ; Suwa1982 ; Swartout1983 ;
Swartout1985 . Although many authors commonly use the terms “explainability”
and “interpretability” as synonyms, Rudin Rudin2019  discusses the problem of
using purely explainable methods to only provide explanations from the results
obtained in black-box models (post-hoc analysis), demystifying the importance
of developing inherently interpretable methodologies with causality
relationships that are understandable to human users.

##  3 Deep Regression Overview

Deep neural networks (DNN) have emerged due to their architecture with
multiple levels of representation and their remarkable performance in a
variety of tasks Bengio2013 . This section aims to discuss DL techniques
commonly employed in regression problems, providing a better understanding and
context for deep fuzzy regression in Section 4.

###  3.1 Convolutional Neural Networks

Convolutional neural networks (CNNs or ConvNets) are feedforward neural
networks with a grid-like topology that are used for applications such as
time-series data processing in 1-D grids and image data processing in 2-D
pixel grids Goodfellow2016 . Figure 2

Figure 2: Convolutional neural network, with 1-D architecture.

presents a CNN architecture for processing time-series data in 1-D grids. The
“neocognitron” model proposed in Fukushima1980  is frequently referred to as
the inspiration model for what is currently known about CNNs. First proposed
in Hubel1968 , the neocognitron aimed to represent simple and complex cells
from the visual cortex of animals which present a mechanism capable of
detecting light in receptive fields Gu2018 .

CNNs use a math operation on two functions called “convolution”, with the
first function referred to as input, the second function as kernel, and the
convolution’s output as the feature map Goodfellow2016 . Outputs from the
convolution layer go through a pooling layer (downsampling), which performs an
overall statistic of the adjacent outputs by reducing the size of the data and
associated parameters via weight sharing Sun2021 ; Li2021 . After the data is
processed along with the layers that alternate between convolution and
pooling, the final feature maps go through a fully connected (dense) layer to
extract high-level features. For regression problems, the extracted features
can be combined in a prediction mechanism with an activation function or a
supervised learning model (e.g., support vector regression) to estimate the
final output Ketkar2017 ; Mallat2016 .

The application of CNNs for regression has been explored for traffic flow
forecasting Wu2022 ; Zhang2020c , prediction of natural environmental factors
Mukhtar2022 ; Heo2021 ; Liu2021c ; Wan2019 , industrial process optimization
Gao2021 ; FAN2020 ; Yuan2020d , electrical/power systems applications
Jalali2021 ; Eskandari2021 ; Zahid2019 ; Koprinska2018 ; Tian2018 , and
chemical process analysis Gao2020 ; Wu2017 . Despite showing good performance
with the extraction of spatially organized features, essentially for pre-
processing, CNN’s performance depends on a large amount of data and the
correct choice of hyperparameters, being computationally intensive
Navamani2019 ; Witten2017 .

###  3.2 Deep Belief Networks

Deep Belief Networks (DBNs) are probabilistic generative models proposed by
Hinton et al. Hinton2006 , which have a hierarchical structure as illustrated
in Figure 3.

Figure 3: Deep Belief Networks architecture.

The DBNs are composed of multiple layers of latent variables (hidden units) of
binary values, organized into multiple learning modules called restricted
Boltzmann machines (RBMs).

Each RBM comprises a layer of visible units for data representation and a
layer of hidden units for feature representation, learned by capturing higher-
order correlations from the data. The two RBM layers, with no connections
within layers, are connected by a matrix of symmetrically weighted
connections, with a total of $L$ weight matrices
$\bm{\mathbf{W}}=\\{\bm{\mathbf{w}}_{1},\bm{\mathbf{w}}_{2},\ldots,\bm{\mathbf{w}}_{L}\\}$,
considering a DBN of $L$ hidden layers. All units in each layer have a
bidirectional connection with all units in neighboring layers, except for the
last two layers, $L$ and output, that have a unidirectional connection
Goodfellow2016 . In the DBN architecture of Figure 3, the layer of visible
units in the first RBM represents the input variables $\bm{\mathbf{x}}$ and
the subsequent layers represent hidden units $\bm{\mathbf{h}}$, progressing
hierarchically until reaching the estimated output, with the output weights
$\bm{\mathbf{w}}_{out}$.

Recent studies with DBNs in the context of regression were mainly applied in
industrial processes for process monitoring Wang2021b ; Hao2021 ; Yuan2021 ;
Yuan2020c ; Hao2020 ; Zhu2019 , soft sensing Wang2021c ; Lian2020 , and
prognostics Tian2020 ; Shao2018 . Other applications include time-series
forecasting Tian2022 ; Xu2019 ; Xie2018 ; Li2017 ; Jia2016  and benchmark
systems modeling Qiao2018 ; Qiu2014 . Studies with DBNs have difficulties in
investigating the influence of hidden units on system dynamics, leading to
interpretability issues. As Figure 3 shows, a DBN needs to undergo
unsupervised and supervised learning, which allows the training process to
become slower as the DBN structure increases. In this way, the DBN becomes
sensitive to noisy inputs by not correctly readjusting its low-level
parameters Salakhutdinov2010 .

###  3.3 Multilayer Autoencoders

Autoencoders (AEs) are feedforward neural networks used for dimensionality
reduction and representation learning, whose training is aimed at mimicking
inputs to outputs Goodfellow2016 . A historical overview of deep learning in
Schmidhuber2015  presented some early works of AEs in the literature, such as
the work in Ballard1987  that proposes unsupervised architectures to
reconstruct the inputs through internal representations. Another early work
was published in Rumelhart1985 , where the authors explore the effect of
hidden units in simple two-layer associative networks, in which they want to
map input patterns to a set of output patterns. As presented in Figure 4, a
single AE essentially has a structure composed of a layer with input variables
$\bm{\mathbf{x}}$, a layer with hidden units $\bm{\mathbf{h}}$ (which performs
an encoding used to represent the input), and an output layer with the
reconstructed inputs represented with a “hat” symbol (e.g.,
$\hat{\bm{\mathbf{x}}}$). These layers are interconnected by an encoder
function (between input and hidden) and by a function decoder (between hidden
and output) Goodfellow2016 . As an evident constraint, the number of neurons
in the input layer must be the same number of neurons in the output layer,
setting AE to unsupervised pre-training or feature extraction Baldi2012 .

Figure 4: Modified Stacked Autoencoder architecture for regression.

A viable alternative to deal with increasingly complex data is to increase the
number of hidden layers in standard AEs, enabling the development of deep
network architectures. A well-known configuration of multilayer autoencoders
found in the literature is stacked autoencoder (SAE). As illustrated in Figure
4, an SAE is developed from the grouping of $L$ AEs, where the hidden layers
of the autoencoders are stacked hierarchically, performing an unsupervised
layerwise learning algorithm. Thus, the reconstruction of inputs after the
$L$-th hidden layer can be disregarded to address regression problems. As for
CNNs, a prediction mechanism or a supervised learning model can be included
after the $L$-th hidden layer to estimate the system output. Related
parameters, such as weights
$\bm{\mathbf{W}}=\\{\bm{\mathbf{w}}_{1},\bm{\mathbf{w}}_{2},\ldots,\bm{\mathbf{w}}_{l}\\}$
between layers, are fine-tuned by a supervised method (e.g., backpropagation
algorithm) Liu2018 .

Recent studies involving types of AEs with deep architecture for regression
applied to various cases of industrial processes, such as soft sensing
Sun2021b ; Sun2020b ; Wang2018b , hydrocracking process Liu2021 ; Yuan2020b ;
Wang2020c , CNC turning machine Shi2019 ; Bose2018 , end-point quality
prediction Zhang2020 ; Liu2020 , and prognostics Wei2022 ; Li2019 ; Ren2018 .
Other works were developed for time-series forecasting Li2022 ; Jin2022 ;
Xiao2021 ; Lv2018 ; Jiao2018 . Some of the limitations of multilayer
autoencoders include the sensitivities to errors or loss of information from
the first layer, impairing learning as it progresses through the hidden
layers. With this, the nature of encoding and decoding by hidden layers can
cause a loss of interpretability and an increase in computational cost
Navamani2019 .

###  3.4 Recurrent Neural Networks

Recurrent Neural Networks (RNNs) are artificial neural networks with an
internal state that allow the use of feedback signals between neurons
Grossberg2013 . One of the early works that culminated in the popularization
of RNNs was proposed in Hopfield1982 , with the development of content-
addressable memory systems called Hopfield networks whose dynamics have a
Lyapunov function (or energy function) to direct to a local minimum of
“energy” associated with the system states Hopfield2007 . Another early work,
published in Rumelhart1986 , presented a new learning procedure,
backpropagation, which adjusts the weights of connections of recurrent
networks with “internal hidden units”. Due to the presence of an “internal
memory”, RNNs are often used to process data in the time domain (sequential
information), with weight sharing through hidden states Zhang2019 . Figure 5

Figure 5: Recurrent Neural Network architecture.

shows the architecture of an RNN, whose hidden states $\bm{\mathbf{h}}$
represent the “internal memory” of the system, and as the inputs are
sequentially observed, the corresponding outputs are estimated. The weights
$\bm{\mathbf{W}}^{in}$, $\bm{\mathbf{W}}^{h}$ and $\bm{\mathbf{W}}^{out}$ are
auxiliary parameters that are shared across time Albertini1998 . The states
are updated at every temporal instant until completing all the input sequences
of length $K$ Pascanu2013 ; Theodoridis2020 . The RNN learning process depends
on a “backpropagation through time” algorithm, which is a gradient-based
technique that updates parameters recursively starting from the last temporal
instant and going backward in time Werbos1990 .

In deep learning, there are many variations of standard RNNs, such as Long
Short-Term Memory networks (LSTMs), Gated Recurrent Units (GRUs), and Echo
State Network (ESNs) Goodfellow2016 . Some of these variations were proposed
to cope with some limitations present in standard RNNs, such as exploding and
vanishing gradients (instability in networks caused by a large variation in
model parameters), overfitting, and difficulty to store low-level features in
long data sequences Lalapura2021 . In this document, only LSTMs and ESNs
employed for regression problems will be discussed.

####  3.4.1 Long Short-Term Memory

Long Short-Term Memory networks (LSTMs) were initially developed in
Hochreiter1997  to overcome the issues of RNNs associated with
vanishing/exploding gradients. These issues can occur during the training of
long temporal sequences with the backpropagation through time algorithm. In
this sense, during successive operations in compound functions with the weight
matrices, gradients can exponentially reach very low values close to zero
(vanishing) or very high values (exploding) Goodfellow2016 . Figure 6
illustrates the architecture of an LSTM.

Figure 6: The Long Short-Term Memory cell.

LSTMs introduced “gates”, nonlinear elements that control memory cells using
sigmoidal functions $\sigma$, hyperbolic tangent functions, current
observation $\bm{\mathbf{x}}(k)$ and hidden units $\bm{\mathbf{h}}(k-1)$ from
the previous time instant Theodoridis2020 . Each of these memory cells has
input, output and forget gates ($\bm{\mathbf{i}}$, $\bm{\mathbf{o}}$ and
$\bm{\mathbf{f}}$, respectively), that protect the information from
perturbations caused by irrelevant inputs and irrelevant memory contents
Witten2017 . The information stored in the cells represents the states
$\bm{\mathbf{s}}(k)$ obtained from the data processed in the time domain.
Despite having the same inputs and outputs as a standard RNN, an LSTM cell has
an internal recurrence (self-loop) to propagate the information flow through a
long sequence and, therefore, more parameters to be adjusted Goodfellow2016 .

Due to the ability to process data sequentially, LSTMs have been used mainly
in applications involving time-series forecasting, such as traffic flow Lu2021
; Yang2019  and natural environment factors Roy2022 ; Kumari2021 ; Altan2021 ;
Ma2020 ; Shen2020 ; Zhang2017 . Other recent LSTM methodologies for regression
have been applied in industrial processes for soft sensing Zhu2021 ; Yuan2021b
; Yuan2020 , process monitoring Cai2020 ; Cheng2019 ; Li2018  and prognostics
Cheng2022 ; Guo2021 , as well as applications involving electrical/power
systems Moradzadeh2020 ; Narayan2017 . Despite the practicality of LSTMs in
regression problems, they can still suffer from vanishing due to the
possibility of saturation of cell states, which must be reset occasionally, in
addition to requiring a high memory bandwidth depending on the execution of
functions inside the cell in complex systems (computational cost issue)
Gers2001 .

####  3.4.2 Echo State Network

Echo State Networks (ESNs) are variations of RNNs that were developed in
Jaeger2001  and share the basic ideas of reservoir computing of Liquid State
Machines from Maass2002 . The term “reservoir computing” stands for a
homonymous research stream that introduced the concept of a dynamic reservoir,
in place of the hidden layer, with many sparsely connected neurons
Schrauwen2007 . In addition, this reservoir must have a stability condition
known as “echo state property”, which allows for a gradual reduction in the
effect of previous states and inputs on future states over time Jaeger2009 .
Figure 7 illustrates the architecture of an ESN.

Figure 7: The Echo State Network.

An ESN induces nonlinear response signals from the input signals
$\bm{\mathbf{x}}(k)$, whose resulting state $\bm{\mathbf{h}}(k)$ echoes the
input information, estimating a desired output signal $\bm{\mathbf{y}}(k)$
Jaeger2001 . In addition to the input weight $\bm{\mathbf{W}}^{in}$ and output
weight $\bm{\mathbf{W}}^{out}$ that are present in a standard RNN, the ESN
features the reservoir weights $\bm{\mathbf{W}}^{r}$ and, optionally, the
output-to-reservoir feedback weights $\bm{\mathbf{W}}^{back}$. The fundamental
idea for the functioning of an ESN is to adjust only $\bm{\mathbf{W}}^{out}$
during training, while the rest are randomly assigned and fixed before
learning Jaeger2007 .

A recurring application of ESN is the time-series prediction in chaotic
systems Tang2022 ; Na2021 ; Yao2019 ; Xu2019b ; McDermott2018 , wind power
systems Lian2022 ; Bai2021 ; Hu2021 ; Tian2021 ; Lopez2018 , solar energy
systems Zhang2021 ; Li2020 ; Xu2018 ; Sun2017 , and benchmark systems Li2022b
; Wang2022 ; Gao2021b . Other applications include industrial processes
Schwedersky2022 ; He2020 ; Huang2020 , electrical/power systems Mansoor2021 ;
Yao2019b ; Wen2019 , and turbofan engine time-series prediction Bala2020 ;
Zhong2017 . It is observed in these studies that the random connectivity
within the dynamic reservoir of an ESN can lead to interpretability issues. In
addition, ESN hyperparameters, such as the number of units in the reservoir
and input scaling, have a short operating space that ensures maximum model
performance Xu2020 .

##  4 Deep Fuzzy Regression

Deep fuzzy systems (DFSs) are models built on top of DL structures with fuzzy
logic systems (FLSs). DFS aims to overcome the lack of interpretability of DL
systems and the limitations of FLS when dealing with high-dimensional data.
DFS is often referred to in the literature as an explainable model due to the
incorporation of fuzzy logic into its core. From the definition from
Gunning2019 , an explainable AI (XAI) system should provide capabilities
accessible to human understanding, reflecting positively on the health of the
system’s processes and being able to operate even in unforeseen situations.
Although DFS is promoted as an XAI system by definition, this is not the case,
as evidenced by works in the literature. This section surveys recent
applications of DFS for regression, with the main focus on its structure and
if it adheres to the XAI principles.

###  4.1 Survey on deep fuzzy systems

This survey will follow two stages to review the DFS for regression
applications. First, the DFS will be categorized according to its structure.
Secondly, the models will be categorized according to whether they follow the
XAI principles. The structures of deep fuzzy models can be represented in
several forms, such as those illustrated in Figure 8. Here, deep fuzzy
structures are summarized into two categories: (i) Standard DFS and (ii)
Hybrid DFS. A model belongs to the first category when the blocks of fuzzy
systems are stacked in series, in parallel, or hierarchically (see Fig. 8a).
Also, there are cases where the architecture of such systems resembles neural
network architecture, such as the dense DNN architecture (see Fig. 8d). The
second category includes hybrid methodologies, where conventional DL models
are combined with FLS. The combination of DL and FLS is commonly in an
ensemble form (see Fig. 8b,c) or mixed form (see Fig. 8d).

Figure 8: Examples of deep fuzzy system frameworks: (a) multiple FLSs
organized hierarchically; (b) sequential ensemble models; (c) parallel
ensemble models; (d) fuzzy neural network fully integrated into a deep
architecture or mixed with parts of a conventional deep model.

Aside from the deep fuzzy structures, the discussed works will be investigated
whether they are in synergy with the XAI principles. If so, these works will
be classified following the categories defined in Angelov2021b : understanding
and scope of the explainability. In terms of understanding, the models will be
classified as (i) transparent or (ii) opaque. Transparent models are those in
which the decisions, predictions, or inner functioning are perceptible or are
visible; they are considered opaque otherwise. The scope is related to
accessing the model’s interpretability through post-hoc explanations,
classified as (i) local, (ii) global, and (iii) visual. Local explanations
facilitate comprehension of small regions of interest in the input space for a
given decision/prediction. Global explanations when such desired understanding
considers the entire sample space. Visual explanations are required when
visual interfaces are needed to demonstrate the influence of features on
decisions.

The following sections will discuss the methods containing DFS for regression
problems. Section 4.1.1 will discuss Standard DFS, while Section 4.1.2 will
discuss Hybrid DFS.

####  4.1.1 Standard deep fuzzy systems

The methods discussed in this section have a DFS structure and were used in
any regression problem domain while adhering to the Standard DFS structure.
They are discussed in order of structural similarity, with structurally
similar methods following each other.

In Huang2021 , a model called Randomly Locally Optimized Deep Fuzzy System
(RLODFS) is proposed. It is composed of a hierarchical structure of a bottom-
up layer-by-layer type with FLS, similar to a fully connected DNN; the
structure of RLODFS is shown in Figure 9. In RLODFS, the input variables are
divided into several fuzzy subsystems, allowing the decrease of the size of
the fuzzy rule’s antecedent, thus reducing the learning complexity when
compared to learning with all input variables at once. The structure of RLODFS
shows several groupings with sharing of input variables performed randomly for
the fuzzy subsystems of level 1, whose outputs are used as inputs of the fuzzy
subsystems of the subsequent levels until reaching the last level fuzzy system
(used to estimate the model output). The training of each fuzzy subsystem was
followed by the Wang-Mendel algorithm Wang1992 , which performs the
construction of fuzzy rules from a small-scale observation set (data pairs)
with input-output mapping. Finally, a random local loop optimization strategy
is performed to remove feature combinations and corresponding subsystems with
low correlation to achieve fast convergence Huang2021 . The performance of the
RLODFS method was compared with DBN, LSTM, generalized regression neural
network (GRNN), among other prediction methods. From the authors’ perspective,
RLODFS has good interpretability due to its structure, clear physical meaning
of its parameters, and the ease of locating fuzzy rules that may fail for
future optimizations. However, there is a lack of transparency in using input
sharing strategies, which increases the method complexity. Furthermore, the
selection of the number of features per fuzzy subsystem needs to be carefully
done, whose manual/arbitrary choice may not reflect the real needs of the case
study under analysis.

Figure 9: The structure of RLODFS based on grouping and input sharing, adapted
from Huang2021 .

The study in Peng2021  proposes a stacked structure composed of double-input
rule modules and interval type-2 fuzzy models, abbreviated as IT2DIRM-DFM. The
proposed model is illustrated in Figure 10, where four layers are presented:
the input layer, which deals directly with the original input data, grouped
two-by-two in each rule module; the stacked layer, where the signals coming
from the input layer become the inputs of the first layer, whose output
becomes the input of the second layer, and so on; the dimension reduction
layer, where the width of the IT2DIRM-DFM is hierarchically reduced until it
becomes two; and the output layer, where the latest IT2DIRM-DFM model produces
the final forecasting results. Still, the authors discuss the interpretability
of the model showing layered learning and fuzzy rules composed of only two
variables in the antecedent, partitioned into interval type-2 second-order
rule partitions. The resulting model was evaluated in two real-world
applications, subway passenger and traffic flow, whose results allowed to
verify the interpretability from consistent readability of which partitions
and bounds of the inputs are used in each fired rule from each rule module.
However, the proposed method is only interpretable locally in each rule module
and not globally, whose structure does not reflect the depth or number of
layers needed to address the experiments. Furthermore, it is not clear the
motivations for considering a function, denoted as f, in each layer related to
the worst-performing module (this is shown at the bottom of Figure 10).

Figure 10: The structure of the double-input rule modules stacked deep
interval type-2 fuzzy model (IT2DIRM-DFM), adapted from Peng2021 .

Another work that explores double-input rule modules within a stacked deep
fuzzy model (in a hierarchical way) was proposed in Li2020b . The authors
investigate the interpretability of the resulting model, called DIRM-DFM, with
conclusions similar to Peng2021 , mainly regarding the composition of fuzzy
rules. However, in DIRM-DFM, they promote more transparency and simplicity.
Some other studies deal with interval type-2 fuzzy models for deep learning in
regression problems. In Qasem2021 , a novel dynamic fractional-order deep
learned type-2 FLS was proposed and constructed using singular value
decomposition and uncertainty bounds type-reduction. In addition to
determining the limit values of the input data (upper and lower singular
values), the authors used stability criteria of fractional-order systems,
allowing to reduce the necessary number of fuzzy rules and reduce the
complexity of nonlinear systems. An evolving recurrent interval type-2
intuitionistic fuzzy neural network (FNN) was proposed in Luo2019  for time-
series prediction. Intuitionistic evaluation, fire strength of membership
degree and strategies for adding and removing fuzzy rules were considered to
improve uncertainty modeling. Both studies in Qasem2021  and Luo2019  did not
present an analysis of the interpretability of their models.

Methods that use multiple neuro-fuzzy systems in a deep hierarchical structure
were proposed in Bodyanskiy2018  and Bodyanskiy2019 . The work in
Bodyanskiy2018  proposed a deep model that cascades multiple neuro-fuzzy
systems modified as multivariable generalized additive models, with
application in real ecological time series. The resulting model manages to
locally detail the mechanisms of each neuro-fuzzy system in each layer.
However, it presents incomplete discussions and results regarding the increase
in the network depth and the influence of the inputs and partial outputs of
the layers on the final prediction. In Bodyanskiy2019 , a hybrid cascade
neuro-fuzzy network was proposed, which is composed of multiple extended neo-
fuzzy neurons with adaptive training designated for online non-stationary data
stream handling. Each layer has a generalization node that performs a weighted
linear combination to obtain an optimal output signal. The experimental
results in electrical loads prediction show the authors’ search for better
accuracy, although at the cost of increasing membership functions to cover the
input space and increasing adjusted parameters (weights).

The work in Cao2021  proposed a deep learned recurrent type-3 fuzzy system
applied for modeling renewable energies (solar panels and wind turbines). The
proposed methodology showed good performance compared to other methods, such
as multilayer perceptron, type-1 FLS, type-2 FLS, and interval type-3 FLS,
despite the lack of a more elaborate discussion of the presented results.
Furthermore, transparency is not guaranteed regarding the modeling steps and
the influence of various parameters optimized during learning. The authors in
Wang2021  proposed a self-organizing FNN with incremental deep pre-training,
abbreviated as IDPT-SOFNN, to promote efficient feature extraction and dynamic
adaptation in the structure according to error-reduction rate. In Wang2020d ,
a deep fuzzy cognitive map was proposed for multivariable time-series
forecasting, with an analysis of the model’s interpretability based on
nonlinear and nonmonotonic influences of unknown exogenous factors. The work
in Park2016  proposed a deep FNN composed of an input layer, four hidden
layers (membership functions, T-norm operation, linear regression, and
aggregation), and an output layer, designed exclusively for intra- and inter-
fractional variational prediction for multiple patients’ breathing motion.

Table 1 summarizes the literature on Standard DFS. They are categorized
according to the application domain and regarding its interpretability
categories. The survey shows that the application domain is vast, with
applications in the domain of industrial systems, power systems, traffic
systems, and multivariable benchmark systems.

Table 1: State-of-the-art on methods with standard deep fuzzy systems for regression problems. XAI: explainable artificial intelligence; Disc.: discussion by the authors (Yes/No); Und.: how understandable is the model, whether it is transparent (T) or opaque (O); Scope: in post-hoc scope, if the model promotes local explanations (L), global explanations (G) or visual explanations (V). Reference |  Approach |  Problem | XAI
---|---|---|---
Disc. | Und. | Scope
Huang2021  |  Hierarchical Randomly Locally Optimized Deep Fuzzy System with input sharing |  Prediction of 12 real-world datasets | Yes | T | L
Peng2021  |  Double-input rule modules stacked deep interval type-2 fuzzy model |  Time-series forecasting | Yes | T | L
Li2020b  |  Double-input-rule-modules stacked deep fuzzy model |  Prediction of short-term photovoltaic power generation | Yes | T | L,G
Qasem2021  |  Dynamic fractional-order deep learned type-2 fuzzy logic system |  Benchmark system prediction | No | O | -
Luo2019  |  Evolving recurrent interval type-2 intuitionistic FNN |  Time-series forecasting | No | T | -
Bodyanskiy2018  |  Deep Stacking Convex Neuro-Fuzzy System |  Real ecological time-series forecasting | No | T | L
Bodyanskiy2019  |  Hybrid cascade neuro-fuzzy scheme (ensemble of extended neo-fuzzy neurons) |  Time-series forecasting (electrical loads) | No | T | -
Cao2021  |  Deep learned recurrent type-3 fuzzy system |  Renewable energy modeling (solar panels and wind turbines) | No | O | -
Wang2021  |  Self-Organizing Deep FNN |  Nonlinear system modeling | No | T | V
Wang2020d  |  Deep Fuzzy cognition map model (with an alternate function gradient descent algorithm) |  Multivariate time-series forecasting | Yes | T | V
Park2016  |  Fuzzy Deep Learning architecture (with four hidden layers) |  Intra- and Inter-fractional variation prediction of Lung Tumors | No | T | L,V

Table 1 shows that only four out of eleven works discuss interpretability,
despite most of their proposed methods being transparent. Also, not all
methods provide post-hoc explanations, and of these, the local scope is more
frequent. The Standard DFS architecture is easy to implement, with flexibility
in the construction of fuzzy rules, having a similar structure to feedforward
neural networks. However, these methods may suffer from loss of
interpretability when using a non-intuitive hierarchical structure, the lack
of investigation of interconnections between the variables involved, and the
limitation of fuzzy rules that can disturb the estimation by not covering all
operating regions (coverage of input data space) Magdalena2019 .

####  4.1.2 Hybrid deep fuzzy systems

The Hybrid DFSs are discussed in this section. The selection and order of
works follow the same principles used for Standard DFS. The common DL
architectures used in combination with fuzzy systems are the Deep Belief
Networks, Autoencoders, Long Short-Term Memory networks, and Echo State
Networks.

In Wang2020 , it is proposed a sparse Deep Belief Network (SDBN) with FNN for
nonlinear system modeling. The SDBN is considered for unsupervised learning
and pre-training to perform fast weight-initialization and improve modeling
robustness. The FNN is used as supervised learning to reduce layer-by-layer
complexity. As shown in Figure 11, the structure of the DBN resembles the
structure shown in Figure 3, except for considering additional constraints
(sparsity) used to penalize fluctuations of values along the hidden neurons.
The proposed method performed better compared to other similar methods, such
as transfer learning-based growing DBN Wang2019 , DBN-based echo-state network
Sun2017 , and self-organizing cascade neural network Li2016 . However, the
authors noticed various fluctuations in the assignment of hyperparameters,
which can compromise the stability of the proposed model, making it necessary
to dynamically and robustly improve its structure to these fluctuations. In
terms of interpretability, the proposed model guarantees a moderate number of
membership functions and rules, allowing good consistency and readability of
what happens within the FNN structure. The same cannot be said for the DBN
structure, which is not intuitive in sparse representation to decide which
features are more valuable than others.

Figure 11: Examples of FLS application with conventional deep models: (a) with
deep belief networks, adapted from Wang2020 ; and (b) with denoising
autoencoders, adapted from Han2020 .

The authors in Han2020  proposed a novel robust deep neural network (RDNN) for
regression problems involving nonlinear systems, with the implementation of
three strategies: a fuzzy denoising autoencoder (FDA) as a base-building unit
for RDNN, improving the ability to represent uncertainties; a compact
parameter strategy (CPS), designed to reconstruct the parameters of the FDA,
reducing unnecessary learning parameters; and an adaptive backpropagation
(ABP) algorithm to update RDNN parameters with fast convergence. As shown in
Figure 11, FDA has an input layer, a fuzzy layer (built by Gaussian membership
functions), and an output layer, which can be divided into an encoder (“input”
to “fuzzy”) and a decoder (“fuzzy” to “output”). Furthermore, at each FDA, due
to the nature of this autoencoder, the data is partially corrupted by noises
(represented with a “tilde” symbol) and is reconstructed using CPS
(represented with a “hat” symbol) with associated parameters (e.g.,
encoder/decoder weights and biases) are adjusted via ABP. Regarding the
interpretability of the proposed model, there was no appropriate discussion by
the authors, as they focused mainly on model performance/accuracy in the
presence of uncertainties. In addition to a fuzzy rule base not being defined,
the number of fuzzy neurons in each FDA is manually initialized and adapts
through redundancies during the reconstruction of parameters via CPS. Finally,
the architecture of the proposed model is not intuitive based on the
mechanisms related to the representation ability of each FDA, which would be
crucial to determine the influence of input variables and learning parameters
on the outputs.

Some hybrid methods using FLS and conventional deep models have been applied
in the literature for traffic flow prediction. In George2020 , the authors
developed an algorithm based on Dolphin Echolocation optimization Borkar2016 ,
where the input features are fuzzified into membership functions to obtain
chronological data, whose integration goes into the weight update process of
the proposed algorithm converging to a globally optimal solution with a Deep
Belief Network. The study in Chen2019b  combined fuzzy information granulation
and a deep neural network to represent the temporal-spatial correlation of
mass traffic data and be able to adapt to noisy data. A Stacked Autoencoder is
used to obtain the prediction results based on processed granules that have a
good capacity for interpretation, which have not been discussed by the
authors.

Other methods were implemented for energy forecasting, with the usual
application of LSTM networks. A novel fuzzy seasonal LSTM was proposed in
Liao2021 , where the fuzzy seasonality index Chang1997  and a decomposition
method were employed to solve the seasonal time-series problem in a monthly
wind power output dataset. In Imani2021 , the authors used an LSTM network
with rough set theory Pawlak1982  and interval type-2 fuzzy sets for short-
term wind speed forecasting, with the aid of mutual information approach for
efficient variable input selection. A novel ultra-short-term photovoltaic
power forecasting method was proposed in Liu2021d , where a T-S fuzzy model is
developed using a fuzzy c-means clustering algorithm and deep belief networks.
The studies in Liao2021 , Imani2021  and Liu2021d  did not discuss the
interpretability of their models, which have a structure with an ensemble
characteristic whose fuzzy part has an affinity for good coverage of the input
space and good capture of data uncertainty.

The work in Chimatapu2020  proposed a deep type-2 FLS (D2FLS) architecture
with greedy layer-wise training for high dimensional input data. The authors
showed how to extract interpretable explanations related to the contribution
of fuzzy rules to the final prediction developed for a two-layer D2FLS.
However, the authors opted for a large number of fuzzy rules (100, in this
case) that impair interpretability by increasing the complexity of the model.
The authors in SN2018  proposed an ensemble model composed of an Echo State
Network (ESN), a T-S fuzzy model, and differential evolution for time-series
forecasting problems. The differential evolution method, used to optimize the
weight coefficients of the model, managed to reduce the number of fuzzy rules.
The interpretability of the model can be impaired due to the structure with an
ensemble characteristic, which does not provide enough transparency for
learning. A method based on the T-S fuzzy model and ESN for the identification
of nonlinear systems was developed in Mahmoud2022 . The authors chose to
balance model complexity and performance based on the parameters involved for
learning (e.g., number of fuzzy rules and reservoir size), which resulted in
better results in comparison with other methods (e.g., traditional ESN and
hybrid fuzzy ESN), despite the lack of discussion on interpretability.

In Wu2020 , a framework based on a sparse autoencoder (SpAE) and a high-order
fuzzy cognitive map (HFCM) is proposed for time-series forecasting problems.
SpAE is used to extract features from the original data, and these features
are via HFCM. Another study that uses SpAE with FLS was proposed in
Sevakula2015  and implemented in time-series forecasting and classification
problems. The proposed method uses a method to reduce fuzzy rules by reducing
the data dimensionality with SpAE. Both studies in Wu2020  and Sevakula2015
do not consider addressing the interpretability of the proposed models, which
present good directions in data partitioning and construction of fuzzy rules
but are not intuitive in feature extraction with sparse representation.

Table 2 summarizes the works presented in this section, in addition to
evaluating them within the context of explainable artificial intelligence
(XAI) systems.

Table 2: State-of-the-art on hybrid methods using fuzzy logic systems and conventional deep models for regression problems. XAI: explainable artificial intelligence; Disc.: discussion by the authors (Yes/No); Und.: how understandable is the model, whether it is transparent (T) or opaque (O); Scope: in post-hoc scope, if the model promotes local explanations (L), global explanations (G) or visual explanations (V). Reference |  Approach |  Problem | XAI
---|---|---|---
Disc. | Und. | Scope
Wang2020  |  Sparse Deep Belief Network with FNN |  Nonlinear system modeling | No | T | -
Han2020  |  Fuzzy denoising autoencoder with a compact parameter strategy and an adaptive backpropagation algorithm |  Prediction of multi-input single-output systems | No | O | -
George2020  |  Chronological Dolphin Echolocation-Fuzzy and Deep Belief Network |  Traffic flow prediction in intelligent transportation system | No | O | -
Chen2019b  |  Fuzzy information granulation with Stacked Autoencoder |  Traffic flow prediction | No | T | -
Liao2021  |  Fuzzy seasonal long short-term memory network |  Wind power forecasting | No | O | -
Imani2021  |  Long Short-Term Memory model hybridized with rough and fuzzy set theory |  Wind speed forecasting | No | O | -
Liu2021d  |  T-S fuzzy model with fuzzy c-means and Deep Belief Network |  Forecasting of photovoltaic power generation | No | T | -
Chimatapu2020  |  Deep Type-2 Fuzzy Logic System, trained as Stacked Autoencoder |  Benchmark system prediction and classification | Yes | T | G,V
SN2018  |  Differential Evolution based Fuzzy Echo State Network |  Time-series forecasting | No | O | -
Mahmoud2022  |  T-S fuzzy model with Echo State Network |  Benchmark system prediction | No | T | -
Wu2020  |  Fuzzy Cognitive Maps with Sparse Autoencoders |  Time-series forecasting | No | T | -
Sevakula2015  |  Fuzzy rule reduction with Stacked Sparse Autoencoders |  Time-series prediction and classification | No | O | -

Only two works discuss the interpretability in Hybrid DFS, half of the
discussed works are opaque, and only one presents post-hoc explanations. This
fact occurs since the combination with DNN brings a new layer of black-box to
the system. However, the Hybrid DFS methods categorized as transparent show
that the fuzzy component can promote good interpretability with efficient
input-output mappings and the construction of rules to cover the universe of
discourse for a given system. Inherently interpretable methodologies are
difficult to achieve only with an ensemble of multiple methods, as seen in
recent studies, in addition to the challenges of reducing the time complexity
of systems, which is slightly mitigated by the reduction of fuzzy rules
Ojha2019 .

##  5 Conclusion

This study surveyed the literature on deep fuzzy systems (DFSs) for regression
applications within the context of explainable artificial intelligence (XAI)
systems (with an emphasis on interpretability). For the survey, the DFSs were
categorized as (i) Standard DFS and (ii) Hybrid DFS. Regarding the
interpretability, each method was categorized as to whether it is transparent
or not and whether it has post-hoc explanations (under the definition of
Angelov2021b ). Standard DFSs have been shown to promote more interpretability
of their models when compared to Hybrid DFSs, according to the survey. Indeed,
Standard DFSs are based on fundamental fuzzy logic systems, which are
inherently interpretable, whereas Hybrid DFSs include conventional deep
learning (DL) methods, which lack flexibility in promoting interpretability.
Furthermore, the DFS is frequently referred to as interpretable by default,
but only 5 of the 23 works surveyed here actually addressed this issue. The
remaining works had a common goal: to improve prediction accuracy using their
proposed methods. However, this survey presented the potential of using
Standard DFS as a base for developing accurate models while promoting
interpretability since hybrid models are not straightforward.

Acknowledgments

This work has been supported by Fundação para a Ciência e a Tecnologia (FCT)
under the project UIDB/00048/2020. Jorge S. S. Júnior is supported by Fundação
para a Ciência e a Tecnologia (FCT) under the grant ref. 2021.04917.BD.

## Competing Interests

The authors declare no competing Interests.

## References

  *   * (1) Busu, M. & Trica, C. L.  Sustainability of Circular Economy Indicators and Their Impact on Economic Growth of the European Union.  _Sustainability_ 11 (19) (2019).  https://doi.org/10.3390/su11195481 .
  * (2) Botev, J., Égert, B. & Jawadi, F.  The nonlinear relationship between economic growth and financial development: Evidence from developing, emerging and advanced economies.  _International Economics_ 160, 3–13 (2019).  https://doi.org/10.1016/j.inteco.2019.06.004 .
  * (3) Liu, B., Zhao, Q., Jin, Y., Shen, J. & Li, C.  Application of combined model of stepwise regression analysis and artificial neural network in data calibration of miniature air quality detector.  _Scientific Reports_ 11 (1), 1–12 (2021).  https://doi.org/10.1038/s41598-021-82871-4 .
  * (4) Gu, K., Qiao, J. & Lin, W.  Recurrent Air Quality Predictor Based on Meteorology- and Pollution-Related Factors.  _IEEE Transactions on Industrial Informatics_ 14 (9), 3946–3955 (2018).  https://doi.org/10.1109/TII.2018.2793950 .
  * (5) Zhu, J. _et al._ Prevalence and influencing factors of anxiety and depression symptoms in the first-line medical staff fighting against COVID-19 in Gansu.  _Frontiers in psychiatry_ 11, 386 (2020).  https://doi.org/10.3389/fpsyt.2020.00386 .
  * (6) Shi, B. _et al._ Nonlinear heart rate variability biomarkers for gastric cancer severity: A pilot study.  _Scientific reports_ 9 (1), 1–9 (2019).  https://doi.org/10.1038/s41598-019-50358-y .
  * (7) Orlandi, M., Escudero-Casao, M. & Licini, G.  Nucleophilicity Prediction via Multivariate Linear Regression Analysis.  _The Journal of Organic Chemistry_ 86 (4), 3555–3564 (2021).  https://doi.org/10.1021/acs.joc.0c02952 .
  * (8) Yang, Y. _et al._ A support vector regression model to predict nitrate-nitrogen isotopic composition using hydro-chemical variables.  _Journal of Environmental Management_ 290, 112674 (2021).  https://doi.org/10.1016/j.jenvman.2021.112674 .
  * (9) Souza, F., Mendes, J. & Araújo, R.  A Regularized Mixture of Linear Experts for Quality Prediction in Multimode and Multiphase Industrial Processes.  _Applied Sciences_ 11 (5) (2021).  https://doi.org/10.3390/app11052040 .
  * (10) Liu, H., Yang, C., Carlsson, B., Qin, S. J. & Yoo, C.  Dynamic Nonlinear Partial Least Squares Modeling Using Gaussian Process Regression.  _Industrial & Engineering Chemistry Research_ 58 (36), 16676–16686 (2019).  https://doi.org/10.1021/acs.iecr.9b00701 .
  * (11) Sun, Q. & Ge, Z.  A Survey on Deep Learning for Data-Driven Soft Sensors.  _IEEE Transactions on Industrial Informatics_ 17 (9), 5853–5866 (2021).  https://doi.org/10.1109/TII.2021.3053128 .
  * (12) Angelov, P. & Soares, E.  Towards explainable deep neural networks (xDNN).  _Neural Networks_ 130, 185–194 (2020).  https://doi.org/10.1016/j.neunet.2020.07.010 .
  * (13) Sjöberg, J. _et al._ Nonlinear black-box modeling in system identification: a unified overview.  _Automatica_ 31 (12), 1691–1724 (1995).  https://doi.org/10.1016/0005-1098(95)00120-8 .
  * (14) Han, Z., Zhao, J., Leung, H., Ma, K. F. & Wang, W.  A Review of Deep Learning Models for Time Series Prediction.  _IEEE Sensors Journal_ 21 (6), 7833–7848 (2021).  https://doi.org/10.1109/JSEN.2019.2923982 .
  * (15) Torres, J. F., Hadjout, D., Sebaa, A., Martínez-Álvarez, F. & Troncoso, A.  Deep Learning for Time Series Forecasting: A Survey.  _Big Data_ 9 (1), 3–21 (2021).  https://doi.org/10.1089/big.2020.0159 .
  * (16) Pang, G., Shen, C., Cao, L. & Hengel, A. V. D.  Deep Learning for Anomaly Detection: A Review.  _ACM Computing Surveys_ 54 (2) (2021).  https://doi.org/10.1145/3439950 .
  * (17) Chalapathy, R. & Chawla, S.  Deep learning for anomaly detection: A survey.  _arXiv preprint arXiv:1901.03407_ (2019).  https://doi.org/10.48550/arXiv.1901.03407 .
  * (18) Dong, S., Wang, P. & Abbas, K.  A survey on deep learning and its applications.  _Computer Science Review_ 40, 100379 (2021).  https://doi.org/10.1016/j.cosrev.2021.100379 .
  * (19) Sengupta, S. _et al._ A review of deep learning with special emphasis on architectures, applications and recent trends.  _Knowledge-Based Systems_ 194, 105596 (2020).  https://doi.org/10.1016/j.knosys.2020.105596 .
  * (20) Pouyanfar, S. _et al._ A Survey on Deep Learning: Algorithms, Techniques, and Applications.  _ACM Computing Surveys_ 51 (5) (2018).  https://doi.org/10.1145/3234150 .
  * (21) Navamani, T.  in Chapter 7 - Efficient Deep Learning Approaches for Health Informatics (ed.Sangaiah, A. K.) _Deep Learning and Parallel Computing Environment for Bioengineering Systems_ 123–137 (Academic Press, 2019).
  * (22) Angelov, P. P., Soares, E. A., Jiang, R., Arnold, N. I. & Atkinson, P. M.  Explainable artificial intelligence: an analytical review.  _WIREs Data Mining and Knowledge Discovery_ (2021).  https://doi.org/10.1002/widm.1424 .
  * (23) Confalonieri, R., Coba, L., Wagner, B. & Besold, T. R.  A historical perspective of explainable Artificial Intelligence.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ 11 (1) (2021).  https://doi.org/10.1002/widm.1391 .
  * (24) Phillips, P. J. _et al._ _Four Principles of Explainable Artificial Intelligence_ (2021).
  * (25) Chimatapu, R., Hagras, H., Starkey, A. & Owusu, G.  _Explainable AI and Fuzzy Logic Systems_ , 3–20 (2018).
  * (26) Łapa, K., Cpałka, K. & Rutkowski, L.  _New Aspects of Interpretability of Fuzzy Systems for Nonlinear Modeling_ , 225–264 (2017).
  * (27) Mendes, J., Maia, R., Araújo, R. & Souza, F. A. A.  Self-Evolving Fuzzy Controller Composed of Univariate Fuzzy Control Rules.  _Applied Sciences_ 10 (17), 5836 (2020).  https://doi.org/10.3390/app10175836 .
  * (28) Moral, J. M. A., Castiello, C., Magdalena, L. & Mencar, C.  _Explainable Fuzzy Systems_ (Springer International Publishing, 2021).
  * (29) Das, R., Sen, S. & Maulik, U.  A Survey on Fuzzy Deep Neural Networks.  _ACM Computing Surveys_ 53 (3) (2020).  https://doi.org/10.1145/3369798 .
  * (30) Zheng, Y., Xu, Z. & Wang, X.  The fusion of deep learning and fuzzy systems: A state-of-the-art survey.  _IEEE Transactions on Fuzzy Systems_ 1–1 (2021).  https://doi.org/10.1109/TFUZZ.2021.3062899 .
  * (31) Zadeh, L. A.  Fuzzy sets.  _Information and Control_ 8 (3), 338 – 353 (1965).  https://doi.org/10.1016/s0019-9958(65)90241-x .
  * (32) Wang, L.-X.  _A Course in Fuzzy Systems and Control_ (Prentice-Hall, Inc., 1997).
  * (33) Mendes, J., Araújo, R., Sousa, P., Apóstolo, F. & Alves, L.  An architecture for adaptive fuzzy control in industrial environments.  _Computers in Industry_ 62 (3), 364–373 (2011).  https://doi.org/10.1016/j.compind.2010.11.001 .
  * (34) Mamdani, E. & Assilian, S.  An experiment in linguistic synthesis with a fuzzy logic controller.  _International Journal of Man-Machine Studies_ 7 (1), 1–13 (1975).  https://doi.org/10.1016/S0020-7373(75)80002-2 .
  * (35) Takagi, T. & Sugeno, M.  Fuzzy identification of systems and its applications to modeling and control.  _IEEE Transactions on Systems, Man, and Cybernetics_ SMC-15 (1), 116–132 (1985).  https://doi.org/10.1109/TSMC.1985.6313399 .
  * (36) Angelov, P. & Yager, R.  (ed.) _Simplified fuzzy rule-based systems using non-parametric antecedents and relative data density_.  (ed.) _2011 IEEE Workshop on Evolving and Adaptive Intelligent Systems (EAIS)_ , 62–69 (2011).
  * (37) Qiu, J., Gao, H. & Ding, S. X.  Recent Advances on Fuzzy-Model-Based Nonlinear Networked Control Systems: A Survey.  _IEEE Transactions on Industrial Electronics_ 63 (2), 1207–1217 (2016).  https://doi.org/10.1109/TIE.2015.2504351 .
  * (38) Ying, H.  (ed.) _General MISO Takagi-Sugeno Fuzzy Systems with Simplified Linear Rule Consequent as Universal Approximators for Control and Modeling Applications_.  (ed.) _IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation_ , Vol. 2, 1335–1340 (1997).
  * (39) Júnior, J. S. S., Mendes, J., Araújo, R., Paulo, J. R. & Premebida, C.  (ed.) _Novelty Detection for Iterative Learning of MIMO Fuzzy Systems_.  (ed.) _2021 IEEE 19th International Conference on Industrial Informatics (INDIN)_ , 1–7 (2021).
  * (40) Hall, P. & Gill, N.  _An introduction to machine learning interpretability_ (O’Reilly Media, Incorporated, 2019).
  * (41) Shortliffe, E. H., Axline, S. G., Buchanan, B. G., Merigan, T. C. & Cohen, S. N.  An Artificial Intelligence program to advise physicians regarding antimicrobial therapy.  _Computers and Biomedical Research_ 6 (6), 544–560 (1973).  https://doi.org/10.1016/0010-4809(73)90029-3 .
  * (42) Clancey, W. J.  Tutoring rules for guiding a case method dialogue.  _International Journal of Man-Machine Studies_ 11 (1), 25–49 (1979).  https://doi.org/10.1016/S0020-7373(79)80004-8 .
  * (43) Weiss, S. M., Kulikowski, C. A., Amarel, S. & Safir, A.  A model-based method for computer-aided medical decision-making.  _Artificial Intelligence_ 11 (1), 145–172 (1978).  https://doi.org/10.1016/0004-3702(78)90015-2 .
  * (44) Suwa, M., Scott, A. C. & Shortliffe, E. H.  An approach to verifying completeness and consistency in a rule-based expert system.  _Ai Magazine_ 3 (4), 16–16 (1982).  https://doi.org/10.1609/aimag.v3i4.377 .
  * (45) Swartout, W. R.  XPLAIN: a system for creating and explaining expert consulting programs.  _Artificial Intelligence_ 21 (3), 285–325 (1983).  https://doi.org/10.1016/S0004-3702(83)80014-9 .
  * (46) Swartout, W. R.  _Explaining and Justifying Expert Consulting Programs_ , 254–271 (1985).
  * (47) Rudin, C.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ 1 (5), 206–215 (2019).  https://doi.org/10.1038/s42256-019-0048-x .
  * (48) Bengio, Y., Courville, A. & Vincent, P.  Representation Learning: A Review and New Perspectives.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 35 (8), 1798–1828 (2013).  https://doi.org/10.1109/TPAMI.2013.50 .
  * (49) Goodfellow, I., Bengio, Y., Courville, A. & Bengio, Y.  _Deep learning_ (MIT press Cambridge, 2016).
  * (50) Fukushima, K.  Neocognitron: A Self Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position.  _Biological Cybernetics_ 36 (4), 193–202 (1980).  https://doi.org/10.1007/bf00344251 .
  * (51) Hubel, D. H. & Wiesel, T. N.  Receptive fields and functional architecture of monkey striate cortex.  _The Journal of Physiology_ 195 (1), 215–243 (1968).  https://doi.org/10.1113/jphysiol.1968.sp008455 .
  * (52) Gu, J. _et al._ Recent advances in convolutional neural networks.  _Pattern Recognition_ 77, 354–377 (2018).  https://doi.org/10.1016/j.patcog.2017.10.013 .
  * (53) Li, Z., Liu, F., Yang, W., Peng, S. & Zhou, J.  A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects.  _IEEE Transactions on Neural Networks and Learning Systems_ 1–21 (2021).  https://doi.org/10.1109/TNNLS.2021.3084827 .
  * (54) Ketkar, N.  _Convolutional Neural Networks_ , 63–78 (2017).
  * (55) Mallat, S.  Understanding deep convolutional networks.  _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ 374 (2065), 20150203 (2016).  https://doi.org/10.1098/rsta.2015.0203 .
  * (56) Wu, S.  Spatiotemporal Dynamic Forecasting and Analysis of Regional Traffic Flow in Urban Road Networks Using Deep Learning Convolutional Neural Network.  _IEEE Transactions on Intelligent Transportation Systems_ 23 (2), 1607–1615 (2022).  https://doi.org/10.1109/TITS.2021.3098461 .
  * (57) Zhang, Y., Zhou, Y., Lu, H. & Fujita, H.  Traffic Network Flow Prediction Using Parallel Training for Deep Convolutional Neural Networks on Spark Cloud.  _IEEE Transactions on Industrial Informatics_ 16 (12), 7369–7380 (2020).  https://doi.org/10.1109/TII.2020.2976053 .
  * (58) Mukhtar, M. _et al._ Development and Comparison of Two Novel Hybrid Neural Network Models for Hourly Solar Radiation Prediction.  _Applied Sciences_ 12 (3) (2022).  https://doi.org/10.3390/app12031435 .
  * (59) Heo, J., Song, K., Han, S. & Lee, D.-E.  Multi-channel convolutional neural network for integration of meteorological and geographical features in solar power forecasting.  _Applied Energy_ 295, 117083 (2021).  https://doi.org/10.1016/j.apenergy.2021.117083 .
  * (60) Liu, T. _et al._ Enhancing Wind Turbine Power Forecast via Convolutional Neural Network.  _Electronics_ 10 (3) (2021).  https://doi.org/10.3390/electronics10030261 .
  * (61) Wan, R., Mei, S., Wang, J., Liu, M. & Yang, F.  Multivariate temporal convolutional network: A deep neural networks approach for multivariate time series forecasting.  _Electronics_ 8 (8) (2019).  https://doi.org/10.3390/electronics8080876 .
  * (62) Gao, Z. _et al._ Multitask-Based Temporal-Channelwise CNN for Parameter Prediction of Two-Phase Flows.  _IEEE Transactions on Industrial Informatics_ 17 (9), 6329–6336 (2021).  https://doi.org/10.1109/TII.2020.2978944 .
  * (63) FAN, W. & ZHANG, Z.  (ed.) _A CNN-SVR Hybrid Prediction Model for Wastewater Index Measurement_.  (ed.) _2020 2nd International Conference on Advances in Computer Technology, Information Science and Communications (CTISC)_ , 90–94 (2020).
  * (64) Yuan, X. _et al._ Soft sensor model for dynamic processes based on multichannel convolutional neural network.  _Chemometrics and Intelligent Laboratory Systems_ 203, 104050 (2020).  https://doi.org/10.1016/j.chemolab.2020.104050 .
  * (65) Jalali, S. M. J. _et al._ A Novel Evolutionary-Based Deep Convolutional Neural Network Model for Intelligent Load Forecasting.  _IEEE Transactions on Industrial Informatics_ 17 (12), 8243–8253 (2021).  https://doi.org/10.1109/TII.2021.3065718 .
  * (66) Eskandari, H., Imani, M. & Moghaddam, M. P.  Convolutional and recurrent neural network based model for short-term load forecasting.  _Electric Power Systems Research_ 195, 107173 (2021).  https://doi.org/10.1016/j.epsr.2021.107173 .
  * (67) Zahid, M. _et al._ Electricity price and load forecasting using enhanced convolutional neural network and enhanced support vector regression in smart grids.  _Electronics_ 8 (2) (2019).  https://doi.org/10.3390/electronics8020122 .
  * (68) Koprinska, I., Wu, D. & Wang, Z.  (ed.) _Convolutional Neural Networks for Energy Time Series Forecasting_.  (ed.) _2018 International Joint Conference on Neural Networks (IJCNN)_ , 1–8 (2018).
  * (69) Tian, C., Ma, J., Zhang, C. & Zhan, P.  A deep neural network model for short-term load forecast based on long short-term memory network and convolutional neural network.  _Energies_ 11 (12) (2018).  https://doi.org/10.3390/en11123493 .
  * (70) Gao, P., Zhang, J., Sun, Y. & Yu, J.  Accurate predictions of aqueous solubility of drug molecules via the multilevel graph convolutional network (MGCN) and SchNet architectures.  _Physical Chemistry Chemical Physics_ 22 (41), 23766–23772 (2020).  https://doi.org/10.1039/D0CP03596C .
  * (71) Wu, K. & Wei, G.-W.  Comparison of multi-task convolutional neural network (MT-CNN) and a few other methods for toxicity prediction.  _arXiv preprint arXiv:1703.10951_ (2017).  https://doi.org/10.48550/arxiv.1703.10951 .
  * (72) Witten, I. H., Frank, E., Hall, M. A. & Pal, C. J.  in Chapter 10 - Deep learning (eds Witten, I. H., Frank, E., Hall, M. A. & Pal, C. J.) _Data Mining (Fourth Edition)_ 417–466 (Morgan Kaufmann, 2017).
  * (73) Hinton, G. E., Osindero, S. & Teh, Y.-W.  A fast learning algorithm for deep belief nets.  _Neural Computation_ 18 (7), 1527–1554 (2006).  https://doi.org/10.1162/neco.2006.18.7.1527 .
  * (74) Wang, Y. _et al._ An Ensemble Deep Belief Network Model Based on Random Subspace for NOx Concentration Prediction.  _ACS Omega_ 6 (11), 7655––7668 (2021).  https://doi.org/10.1021/acsomega.0c06317 .
  * (75) Hao, X. _et al._ Prediction of nitrogen oxide emission concentration in cement production process: a method of deep belief network with clustering and time series.  _Environmental Science and Pollution Research_ 28 (24), 31689––31703 (2021).  https://doi.org/10.1007/s11356-021-12834-9 .
  * (76) Yuan, X., Gu, Y. & Wang, Y.  Supervised Deep Belief Network for Quality Prediction in Industrial Processes.  _IEEE Transactions on Instrumentation and Measurement_ 70, 1–11 (2021).  https://doi.org/10.1109/TIM.2020.3035464 .
  * (77) Yuan, X. _et al._ FeO Content Prediction for an Industrial Sintering Process based on Supervised Deep Belief Network.  _IFAC-PapersOnLine_ 53 (2), 11883–11888 (2020).  https://doi.org/10.1016/j.ifacol.2020.12.703 .
  * (78) Hao, X. _et al._ Energy consumption prediction in cement calcination process: A method of deep belief network with sliding window.  _Energy_ 207, 118256 (2020).  https://doi.org/10.1016/j.energy.2020.118256 .
  * (79) Zhu, S.-B., Li, Z.-L., Zhang, S.-M., Ying-Yu & Zhang, H.-F.  Deep belief network-based internal valve leakage rate prediction approach.  _Measurement_ 133, 182–192 (2019).  https://doi.org/10.1016/j.measurement.2018.10.020 .
  * (80) Wang, G., Jia, Q.-S., Zhou, M., Bi, J. & Qiao, J.  Soft-sensing of Wastewater Treatment Process via Deep Belief Network with Event-triggered Learning.  _Neurocomputing_ 436, 103–113 (2021).  https://doi.org/10.1016/j.neucom.2020.12.108 .
  * (81) Lian, P., Liu, H., Wang, X. & Guo, R.  Soft sensor based on DBN-IPSO-SVR approach for rotor thermal deformation prediction of rotary air-preheater.  _Measurement_ 165, 108109 (2020).  https://doi.org/10.1016/j.measurement.2020.108109 .
  * (82) Tian, W., Liu, Z., Li, L., Zhang, S. & Li, C.  Identification of abnormal conditions in high-dimensional chemical process based on feature selection and deep learning.  _Chinese Journal of Chemical Engineering_ 28 (7), 1875–1883 (2020).  https://doi.org/10.1016/j.cjche.2020.05.003 .
  * (83) Shao, H., Jiang, H., Li, X. & Liang, T.  Rolling bearing fault detection using continuous deep belief network with locally linear embedding.  _Computers in Industry_ 96, 27–39 (2018).  https://doi.org/10.1016/j.compind.2018.01.005 .
  * (84) Tian, J., Liu, Y., Zheng, W. & Yin, L.  Smog prediction based on the deep belief - BP neural network model (DBN-BP).  _Urban Climate_ 41, 101078 (2022).  https://doi.org/10.1016/j.uclim.2021.101078 .
  * (85) Xu, H. & Jiang, C.  Deep belief network-based support vector regression method for traffic flow forecasting.  _Neural Computing and Applications_ 32 (7), 2027––2036 (2019).  https://doi.org/10.1007/s00521-019-04339-x .
  * (86) Xie, T., Zhang, G., Liu, H., Liu, F. & Du, P.  A Hybrid Forecasting Method for Solar Output Power Based on Variational Mode Decomposition, Deep Belief Networks and Auto-Regressive Moving Average.  _Applied Sciences_ 8 (10) (2018).  https://doi.org/10.3390/app8101901 .
  * (87) Li, X., Yang, L., Xue, F. & Zhou, H.  (ed.) _Time series prediction of stock price using deep belief networks with intrinsic plasticity_.  (ed.) _2017 29th Chinese Control And Decision Conference (CCDC)_ , 1237–1242 (2017).
  * (88) Jia, Y., Wu, J. & Du, Y.  (ed.) _Traffic speed prediction using deep learning method_.  (ed.) _2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC)_ , 1217–1222 (2016).
  * (89) Qiao, J., Wang, G., Li, W. & Li, X.  A deep belief network with PLSR for nonlinear system modeling.  _Neural Networks_ 104, 68–79 (2018).  https://doi.org/10.1016/j.neunet.2017.10.006 .
  * (90) Qiu, X., Zhang, L., Ren, Y., Suganthan, P. N. & Amaratunga, G.  (ed.) _Ensemble deep learning for regression and time series forecasting_.  (ed.) _2014 IEEE Symposium on Computational Intelligence in Ensemble Learning (CIEL)_ , 1–6 (2014).
  * (91) Salakhutdinov, R. & Larochelle, H.  (ed.) _Efficient learning of deep Boltzmann machines_.  (ed.) _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , 693–700 (2010).
  * (92) Schmidhuber, J.  Deep learning in neural networks: An overview.  _Neural Networks_ 61, 85–117 (2015).  https://doi.org/10.1016/j.neunet.2014.09.003 .
  * (93) Ballard, D. H.  (ed.) _Modular learning in neural networks_.  (ed.) _Proceedings of the Sixth National Conference on Artificial Intelligence - Volume 1_ , Vol. 647, 279–284 (1987).
  * (94) Rumelhart, D. E., Hinton, G. E. & Williams, R. J.  Learning Internal Representations by Error Propagation.  Tech. Rep., California Univ., San Diego, La Jolla. Inst. for Cognitive Science (1985).
  * (95) Baldi, P.  (ed.) _Autoencoders, Unsupervised Learning, and Deep Architectures_.  (ed.) _Proceedings of ICML Workshop on Unsupervised and Transfer Learning_ , Vol. 27, 37–49 (2012).
  * (96) Liu, G., Bao, H. & Han, B.  A Stacked Autoencoder-Based Deep Neural Network for Achieving Gearbox Fault Diagnosis.  _Mathematical Problems in Engineering_ 2018, 1–10 (2018).  https://doi.org/10.1155/2018/5105709 .
  * (97) Sun, Q. & Ge, Z.  Deep Learning for Industrial KPI Prediction: When Ensemble Learning Meets Semi-Supervised Data.  _IEEE Transactions on Industrial Informatics_ 17 (1), 260–269 (2021).  https://doi.org/10.1109/TII.2020.2969709 .
  * (98) Sun, Q. & Ge, Z.  Gated Stacked Target-Related Autoencoder: A Novel Deep Feature Extraction and Layerwise Ensemble Method for Industrial Soft Sensor Application.  _IEEE Transactions on Cybernetics_ 1–12 (2020).  https://doi.org/10.1109/TCYB.2020.3010331 .
  * (99) Wang, X. & Liu, H.  Soft sensor based on stacked auto-encoder deep neural network for air preheater rotor deformation prediction.  _Advanced Engineering Informatics_ 36, 112–119 (2018).  https://doi.org/10.1016/j.aei.2018.03.003 .
  * (100) Liu, C., Wang, Y., Wang, K. & Yuan, X.  Deep learning with nonlocal and local structure preserving stacked autoencoder for soft sensor in industrial processes.  _Engineering Applications of Artificial Intelligence_ 104, 104341 (2021).  https://doi.org/10.1016/j.engappai.2021.104341 .
  * (101) Yuan, X., Ou, C., Wang, Y., Yang, C. & Gui, W.  A novel semi-supervised pre-training strategy for deep networks and its application for quality variable prediction in industrial processes.  _Chemical Engineering Science_ 217, 115509 (2020).  https://doi.org/10.1016/j.ces.2020.115509 .
  * (102) Wang, Y., Liu, C. & Yuan, X.  Stacked locality preserving autoencoder for feature extraction and its application for industrial process data modeling.  _Chemometrics and Intelligent Laboratory Systems_ 203, 104086 (2020).  https://doi.org/10.1016/j.chemolab.2020.104086 .
  * (103) Shi, C. _et al._ Using Multiple-Feature-Spaces-Based Deep Learning for Tool Condition Monitoring in Ultraprecision Manufacturing.  _IEEE Transactions on Industrial Electronics_ 66 (5), 3794–3803 (2019).  https://doi.org/10.1109/TIE.2018.2856193 .
  * (104) Bose, T., Majumdar, A. & Chattopadhyay, T.  (ed.) _Machine Load Estimation Via Stacked Autoencoder Regression_.  (ed.) _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2126–2130 (2018).
  * (105) Zhang, X., Zou, Y. & Li, S.  Enhancing incremental deep learning for FCCU end-point quality prediction.  _Information Sciences_ 530, 95–107 (2020).  https://doi.org/10.1016/j.ins.2020.04.013 .
  * (106) Liu, C., Tang, L. & Liu, J.  A Stacked Autoencoder With Sparse Bayesian Regression for End-Point Prediction Problems in Steelmaking Process.  _IEEE Transactions on Automation Science and Engineering_ 17 (2), 550–561 (2020).  https://doi.org/10.1109/TASE.2019.2935314 .
  * (107) Wei, M., Ye, M., Wang, Q., Xinxin-Xu & Twajamahoro, J. P.  Remaining useful life prediction of lithium-ion batteries based on stacked autoencoder and gaussian mixture regression.  _Journal of Energy Storage_ 47, 103558 (2022).  https://doi.org/10.1016/j.est.2021.103558 .
  * (108) Li, Z., Li, J., Wang, Y. & Wang, K.  A deep learning approach for anomaly detection based on SAE and LSTM in mechanical equipment.  _The International Journal of Advanced Manufacturing Technology_ 103 (1–-4), 499––510 (2019).  https://doi.org/10.1007/s00170-019-03557-w .
  * (109) Ren, L., Sun, Y., Cui, J. & Zhang, L.  Bearing remaining useful life prediction based on deep autoencoder and deep neural networks.  _Journal of Manufacturing Systems_ 48, 71–77 (2018).  https://doi.org/10.1016/j.jmsy.2018.04.008 .
  * (110) Li, M., Xie, X. & Zhang, D.  Improved Deep Learning Model Based on Self-Paced Learning for Multiscale Short-Term Electricity Load Forecasting.  _Sustainability_ 14 (1) (2022).  https://doi.org/10.3390/su14010188 .
  * (111) Jin, X.-B., Gong, W.-T., Kong, J.-L., Bai, Y.-T. & Su, T.-L.  PFVAE: A Planar Flow-Based Variational Auto-Encoder Prediction Model for Time Series Data.  _Mathematics_ 10 (4) (2022).  https://doi.org/10.3390/math10040610 .
  * (112) Xiao, X. _et al._ SSAE‐MLP: Stacked sparse autoencoders‐based multi‐layer perceptron for main bearing temperature prediction of large‐scale wind turbines.  _Concurrency and Computation: Practice and Experience_ 33 (17) (2021).  https://doi.org/10.1002/cpe.6315 .
  * (113) Lv, S.-X., Peng, L. & Wang, L.  Stacked autoencoder with echo-state regression for tourism demand forecasting using search query data.  _Applied Soft Computing_ 73, 119–133 (2018).  https://doi.org/10.1016/j.asoc.2018.08.024 .
  * (114) Jiao, R., Huang, X., Ma, X., Han, L. & Tian, W.  A Model Combining Stacked Auto Encoder and Back Propagation Algorithm for Short-Term Wind Power Forecasting.  _IEEE Access_ 6, 17851–17858 (2018).  https://doi.org/10.1109/ACCESS.2018.2818108 .
  * (115) Grossberg, S.  Recurrent neural networks.  _Scholarpedia_ 8 (2), 1888 (2013).  https://doi.org/10.4249/scholarpedia.1888 .
  * (116) Hopfield, J. J.  Neural networks and physical systems with emergent collective computational abilities.  _Proceedings of the National Academy of Sciences_ 79 (8), 2554–2558 (1982).  https://doi.org/10.1073/pnas.79.8.2554 .
  * (117) Hopfield, J. J.  Hopfield network.  _Scholarpedia_ 2 (5), 1977 (2007).  https://doi.org/10.4249/scholarpedia.1977 .
  * (118) Rumelhart, D. E., Hinton, G. E. & Williams, R. J.  Learning representations by back-propagating errors.  _Nature_ 323 (6088), 533––536 (1986).  https://doi.org/10.1038/323533a0 .
  * (119) Zhang, S., Bamakan, S. M. H., Qu, Q. & Li, S.  Learning for Personalized Medicine: A Comprehensive Review From a Deep Learning Perspective.  _IEEE Reviews in Biomedical Engineering_ 12, 194–208 (2019).  https://doi.org/10.1109/RBME.2018.2864254 .
  * (120) Albertini, F. & Pra, P. D.  in Recurrent neural networks: Identification and other system theoretic properties (ed.) _Neural Network Systems Techniques and Applications_ , Vol. 3 1–49 (1998).
  * (121) Pascanu, R., Gulcehre, C., Cho, K. & Bengio, Y.  How to Construct Deep Recurrent Neural Networks.  _arXiv preprint arXiv:1312.6026_ (2013).  https://doi.org/10.48550/ARXIV.1312.6026 .
  * (122) Theodoridis, S.  in Chapter 18 - Neural Networks and Deep Learning (ed.) _Machine Learning (Second Edition)_ 901–1038 (2020).
  * (123) Werbos, P.  Backpropagation through time: what it does and how to do it.  _Proceedings of the IEEE_ 78 (10), 1550–1560 (1990).  https://doi.org/10.1109/5.58337 .
  * (124) Lalapura, V. S., Amudha, J. & Satheesh, H. S.  Recurrent Neural Networks for Edge Intelligence: A Survey.  _ACM Computing Surveys_ 54 (4), 1–38 (2021).  https://doi.org/10.1145/3448974 .
  * (125) Hochreiter, S. & Schmidhuber, J.  Long Short-Term Memory.  _Neural Computation_ 9 (8), 1735–1780 (1997).  https://doi.org/10.1162/neco.1997.9.8.1735 .
  * (126) Lu, S., Zhang, Q., Chen, G. & Seng, D.  A combined method for short-term traffic flow prediction based on recurrent neural network.  _Alexandria Engineering Journal_ 60 (1), 87–94 (2021).  https://doi.org/10.1016/j.aej.2020.06.008 .
  * (127) Yang, B., Sun, S., Li, J., Lin, X. & Tian, Y.  Traffic flow prediction using LSTM with feature enhancement.  _Neurocomputing_ 332, 320–327 (2019).  https://doi.org/10.1016/j.neucom.2018.12.016 .
  * (128) Roy, D. K. _et al._ Daily Prediction and Multi-Step Forward Forecasting of Reference Evapotranspiration Using LSTM and Bi-LSTM Models.  _Agronomy_ 12 (3) (2022).  https://doi.org/10.3390/agronomy12030594 .
  * (129) Kumari, P. & Toshniwal, D.  Long short term memory–convolutional neural network based deep hybrid approach for solar irradiance forecasting.  _Applied Energy_ 295, 117061 (2021).  https://doi.org/10.1016/j.apenergy.2021.117061 .
  * (130) Altan, A., Karasu, S. & Zio, E.  A new hybrid model for wind speed forecasting combining long short-term memory neural network, decomposition methods and grey wolf optimizer.  _Applied Soft Computing_ 100, 106996 (2021).  https://doi.org/10.1016/j.asoc.2020.106996 .
  * (131) Ma, J. _et al._ Air quality prediction at new stations using spatially transferred bi-directional long short-term memory network.  _Science of The Total Environment_ 705, 135771 (2020).  https://doi.org/10.1016/j.scitotenv.2019.135771 .
  * (132) Shen, Z., Zhang, Y., Lu, J., Xu, J. & Xiao, G.  A novel time series forecasting model with deep learning.  _Neurocomputing_ 396, 302–313 (2020).  https://doi.org/10.1016/j.neucom.2018.12.084 .
  * (133) Zhang, Q., Wang, H., Dong, J., Zhong, G. & Sun, X.  Prediction of Sea Surface Temperature Using Long Short-Term Memory.  _IEEE Geoscience and Remote Sensing Letters_ 14 (10), 1745–1749 (2017).  https://doi.org/10.1109/LGRS.2017.2733548 .
  * (134) Zhu, X., Hao, K., Xie, R. & Huang, B.  Soft sensor based on extreme gradient boosting and bidirectional converted gates long short-term memory self-attention network.  _Neurocomputing_ 434, 126–136 (2021).  https://doi.org/10.1016/j.neucom.2020.12.028 .
  * (135) Yuan, X., Li, L., Shardt, Y. A. W., Wang, Y. & Yang, C.  Deep Learning With Spatiotemporal Attention-Based LSTM for Industrial Soft Sensor Model Development.  _IEEE Transactions on Industrial Electronics_ 68 (5), 4404–4414 (2021).  https://doi.org/10.1109/TIE.2020.2984443 .
  * (136) Yuan, X., Li, L. & Wang, Y.  Nonlinear Dynamic Soft Sensor Modeling With Supervised Long Short-Term Memory Network.  _IEEE Transactions on Industrial Informatics_ 16 (5), 3168–3176 (2020).  https://doi.org/10.1109/TII.2019.2902129 .
  * (137) Cai, W., Zhang, W., Hu, X. & Liu, Y.  A hybrid information model based on long short-term memory network for tool condition monitoring.  _Journal of Intelligent Manufacturing_ 31 (6), 1497––1510 (2020).  https://doi.org/10.1007/s10845-019-01526-4 .
  * (138) Cheng, Y., Zhu, H., Wu, J. & Shao, X.  Machine Health Monitoring Using Adaptive Kernel Spectral Clustering and Deep Long Short-Term Memory Recurrent Neural Networks.  _IEEE Transactions on Industrial Informatics_ 15 (2), 987–997 (2019).  https://doi.org/10.1109/TII.2018.2866549 .
  * (139) Li, S., Fang, H. & Shi, B.  (ed.) _Multi-Step-Ahead Prediction with Long Short Term Memory Networks and Support Vector Regression_.  (ed.) _2018 37th Chinese Control Conference (CCC)_ , 8104–8109 (2018).
  * (140) Cheng, Y., Wang, C., Wu, J., Zhu, H. & Lee, C.  Multi-dimensional recurrent neural network for remaining useful life prediction under variable operating conditions and multiple fault modes.  _Applied Soft Computing_ 118, 108507 (2022).  https://doi.org/10.1016/j.asoc.2022.108507 .
  * (141) Guo, J., Lao, Z., Hou, M., Li, C. & Zhang, S.  Mechanical fault time series prediction by using EFMSAE-LSTM neural network.  _Measurement_ 173, 108566 (2021).  https://doi.org/10.1016/j.measurement.2020.108566 .
  * (142) Moradzadeh, A., Zakeri, S., Shoaran, M., Mohammadi-Ivatloo, B. & Mohammadi, F.  Short-Term Load Forecasting of Microgrid via Hybrid Support Vector Regression and Long Short-Term Memory Algorithms.  _Sustainability_ 12 (17) (2020).  https://doi.org/10.3390/su12177076 .
  * (143) Narayan, A. & Hipel, K. W.  (ed.) _Long short term memory networks for short-term electric load forecasting_.  (ed.) _2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ , 2573–2578 (2017).
  * (144) Gers, F.  _Long short-term memory in recurrent neural networks_.  Ph.D. thesis, École Polytechnique Fédérale de Lausanne (2001).
  * (145) Jaeger, H.  The “echo state” approach to analysing and training recurrent neural networks-with an erratum note.  Tech. Rep. 34, Bonn, Germany: German National Research Center for Information Technology GMD Technical Report (2001).
  * (146) Maass, W., Natschläger, T. & Markram, H.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations.  _Neural Computation_ 14 (11), 2531–2560 (2002).  https://doi.org/10.1162/089976602760407955 .
  * (147) Schrauwen, B., Verstraeten, D. & Campenhout, J. V.  (ed.) _An overview of reservoir computing: theory, applications and implementations_.  (ed.) _Proceedings of the 15th European Symposium on Artificial Neural Networks_ , 471–482 (2007).
  * (148) Lukoševičius, M. & Jaeger, H.  Reservoir computing approaches to recurrent neural network training.  _Computer Science Review_ 3 (3), 127–149 (2009).  https://doi.org/10.1016/j.cosrev.2009.03.005 .
  * (149) Jaeger, H.  Echo state network.  _Scholarpedia_ 2 (9), 2330 (2007).  https://doi.org/10.4249/scholarpedia.2330 .
  * (150) Tang, Y., Deng, J., Zang, C. & Wu, Q.  Chaotic Modeling of Stream Nitrate Concentration and Transportation via IFPA-ESN and Turning Point Analyses.  _Frontiers in Environmental Science_ 10 (2022).  https://doi.org/10.3389/fenvs.2022.855694 .
  * (151) Na, X., Ren, W. & Xu, X.  Hierarchical delay-memory echo state network: A model designed for multi-step chaotic time series prediction.  _Engineering Applications of Artificial Intelligence_ 102, 104229 (2021).  https://doi.org/10.1016/j.engappai.2021.104229 .
  * (152) Yao, X., Wang, Z. & Zhang, H.  Prediction and identification of discrete-time dynamic nonlinear systems based on adaptive echo state network.  _Neural Networks_ 113, 11–19 (2019).  https://doi.org/10.1016/j.neunet.2019.01.003 .
  * (153) Xu, M., Han, M., Qiu, T. & Lin, H.  Hybrid Regularized Echo State Network for Multivariate Chaotic Time Series Prediction.  _IEEE Transactions on Cybernetics_ 49 (6), 2305–2315 (2019).  https://doi.org/10.1109/TCYB.2018.2825253 .
  * (154) McDermott, P. L. & Wikle, C. K.  Deep echo state networks with uncertainty quantification for spatio‐temporal forecasting.  _Environmetrics_ 30 (3) (2018).  https://doi.org/10.1002/env.2553 .
  * (155) Lian, L.  Wind speed prediction based on CEEMD-SE and multiple echo state network with Gauss–Markov fusion.  _Review of Scientific Instruments_ 93 (1), 015105 (2022).  https://doi.org/10.1063/5.0081086 .
  * (156) Bai, Y., Liu, M.-D., Ding, L. & Ma, Y.-J.  Double-layer staged training echo-state networks for wind speed prediction using variational mode decomposition.  _Applied Energy_ 301, 117461 (2021).  https://doi.org/10.1016/j.apenergy.2021.117461 .
  * (157) Hu, H., Wang, L. & Tao, R.  Wind speed forecasting based on variational mode decomposition and improved echo state network.  _Renewable Energy_ 164, 729–751 (2021).  https://doi.org/10.1016/j.renene.2020.09.109 .
  * (158) Tian, Z.  Approach for short-term wind power prediction via kernel principal component analysis and echo state network optimized by improved particle swarm optimization algorithm.  _Transactions of the Institute of Measurement and Control_ 43 (16), 3647–3662 (2021).  https://doi.org/10.1177/01423312211046421 .
  * (159) López, E., Valle, C., Allende, H., Gil, E. & Madsen, H.  Wind Power Forecasting Based on Echo State Networks and Long Short-Term Memory.  _Energies_ 11 (3) (2018).  https://doi.org/10.3390/en11030526 .
  * (160) Zhang, H. _et al._ Self-organizing deep belief modular echo state network for time series prediction.  _Knowledge-Based Systems_ 222, 107007 (2021).  https://doi.org/10.1016/j.knosys.2021.107007 .
  * (161) Li, Q., Wu, Z. & Zhang, H.  Spatio-temporal modeling with enhanced flexibility and robustness of solar irradiance prediction: A chain-structure echo state network approach.  _Journal of Cleaner Production_ 261, 121151 (2020).  https://doi.org/10.1016/j.jclepro.2020.121151 .
  * (162) Xu, M., Han, M. & Lin, H.  Wavelet-denoising multiple echo state networks for multivariate time series prediction.  _Information Sciences_ 465, 439–458 (2018).  https://doi.org/10.1016/j.ins.2018.07.015 .
  * (163) Sun, X., Li, T., Li, Q., Huang, Y. & Li, Y.  Deep belief echo-state network and its application to time series prediction.  _Knowledge-Based Systems_ 130, 17–29 (2017).  https://doi.org/10.1016/j.knosys.2017.05.022 .
  * (164) Li, Z. & Tanaka, G.  Multi-reservoir echo state networks with sequence resampling for nonlinear time-series prediction.  _Neurocomputing_ 467, 115–129 (2022).  https://doi.org/10.1016/j.neucom.2021.08.122 .
  * (165) Wang, L., Su, Z., Qiao, J. & Deng, F.  A pseudo-inverse decomposition-based self-organizing modular echo state network for time series prediction.  _Applied Soft Computing_ 116, 108317 (2022).  https://doi.org/10.1016/j.asoc.2021.108317 .
  * (166) Gao, R., Du, L., Duru, O. & Yuen, K. F.  Time series forecasting based on echo state network and empirical wavelet transformation.  _Applied Soft Computing_ 102, 107111 (2021).  https://doi.org/10.1016/j.asoc.2021.107111 .
  * (167) Schwedersky, B. B., Flesch, R. C. C. & Dangui, H. A. S.  Nonlinear MIMO System Identification with Echo-State Networks.  _Journal of Control, Automation and Electrical Systems_ 1–12 (2022).  https://doi.org/10.1007/s40313-021-00874-y .
  * (168) He, Y.-L., Tian, Y., Xu, Y. & Zhu, Q.-X.  Novel soft sensor development using echo state network integrated with singular value decomposition: Application to complex chemical processes.  _Chemometrics and Intelligent Laboratory Systems_ 200, 103981 (2020).  https://doi.org/10.1016/j.chemolab.2020.103981 .
  * (169) Huang, R., Li, Z. & Cao, B.  A Soft Sensor Approach Based on an Echo State Network Optimized by Improved Genetic Algorithm.  _Sensors_ 20 (17) (2020).  https://doi.org/10.3390/s20175000 .
  * (170) Mansoor, M., Grimaccia, F., Leva, S. & Mussetta, M.  Comparison of echo state network and feed-forward neural networks in electrical load forecasting for demand response programs.  _Mathematics and Computers in Simulation_ 184, 282–293 (2021).  https://doi.org/10.1016/j.matcom.2020.07.011 .
  * (171) Yao, X., Wang, Z. & Zhang, H.  A novel photovoltaic power forecasting model based on echo state network.  _Neurocomputing_ 325, 182–189 (2019).  https://doi.org/10.1016/j.neucom.2018.10.022 .
  * (172) Wen, S. _et al._ Memristor-Based Echo State Network With Online Least Mean Square.  _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ 49 (9), 1787–1796 (2019).  https://doi.org/10.1109/TSMC.2018.2825021 .
  * (173) Bala, A., Ismail, I., Ibrahim, R., Sait, S. M. & Oliva, D.  An Improved Grasshopper Optimization Algorithm Based Echo State Network for Predicting Faults in Airplane Engines.  _IEEE Access_ 8, 159773–159789 (2020).  https://doi.org/10.1109/ACCESS.2020.3020356 .
  * (174) Zhong, S., Xie, X., Lin, L. & Wang, F.  Genetic algorithm optimized double-reservoir echo state network for multi-regime time series prediction.  _Neurocomputing_ 238, 191–204 (2017).  https://doi.org/10.1016/j.neucom.2017.01.053 .
  * (175) Xu, Y.  A Review of Machine Learning With Echo State Networks.  _Project Report_ (2020) .
  * (176) Gunning, D. _et al._ XAI—Explainable artificial intelligence.  _Science Robotics_ 4 (37) (2019).  https://doi.org/10.1126/scirobotics.aay7120 .
  * (177) Huang, Y., Chen, D., Zhao, W. & Mo, H.  Deep Fuzzy System Algorithms Based on Deep Learning and Input Sharing for Regression Application.  _International Journal of Fuzzy Systems_ 23 (3), 727–742 (2021).  https://doi.org/10.1007/s40815-020-00998-4 .
  * (178) Wang, L.-X. & Mendel, J.  Generating fuzzy rules by learning from examples.  _IEEE Transactions on Systems, Man, and Cybernetics_ 22 (6), 1414–1427 (1992).  https://doi.org/10.1109/21.199466 .
  * (179) Peng, W., Zhou, C., Li, C., Deng, X. & Zhang, G.  Double-Input Rule Modules Stacked Deep Interval Type-2 Fuzzy Model with Application to Time Series Forecasting.  _International Journal of Fuzzy Systems_ 23 (5), 1326–1346 (2021).  https://doi.org/10.1007/s40815-021-01087-w .
  * (180) Li, C., Zhou, C., Peng, W., Lv, Y. & Luo, X.  Accurate prediction of short-term photovoltaic power generation via a novel double-input-rule-modules stacked deep fuzzy method.  _Energy_ 212, 118700 (2020).  https://doi.org/10.1016/j.energy.2020.118700 .
  * (181) Qasem, S. N. & Mohammadzadeh, A.  A deep learned type-2 fuzzy neural network: Singular value decomposition approach.  _Applied Soft Computing_ 105, 107244 (2021).  https://doi.org/10.1016/j.asoc.2021.107244 .
  * (182) Luo, C., Tan, C., Wang, X. & Zheng, Y.  An evolving recurrent interval type-2 intuitionistic fuzzy neural network for online learning and time series prediction.  _Applied Soft Computing_ 78, 150–163 (2019).  https://doi.org/10.1016/j.asoc.2019.02.032 .
  * (183) Bodyanskiy, Y., Vynokurova, O., Pliss, I., Peleshko, D. & Rashkevych, Y.  (ed.) _Deep Stacking Convex Neuro-Fuzzy System and Its On-line Learning_.  (ed.) _DepCoS-RELCOMEX 2017: Advances in Dependability Engineering of Complex Systems_ , 49––59 (2018).
  * (184) Bodyanskiy, Y. V. & Tyshchenko, O. K.  A Hybrid Cascade Neuro–Fuzzy Network with Pools of Extended Neo–Fuzzy Neurons and its Deep Learning.  _International Journal of Applied Mathematics and Computer Science_ 29 (3), 477––488 (2019).  https://doi.org/10.2478/amcs-2019-0035 .
  * (185) Cao, Y. _et al._ Deep learned recurrent type-3 fuzzy system: Application for renewable energy modeling/prediction.  _Energy Reports_ 7, 8115–8127 (2021).  https://doi.org/10.1016/j.egyr.2021.07.004 .
  * (186) Wang, G. & Qiao, J.  An Efficient Self-Organizing Deep Fuzzy Neural Network for Nonlinear System Modeling.  _IEEE Transactions on Fuzzy Systems_ 1–1 (2021).  https://doi.org/10.1109/TFUZZ.2021.3077396 .
  * (187) Wang, J., Peng, Z., Wang, X., Li, C. & Wu, J.  Deep Fuzzy Cognitive Maps for Interpretable Multivariate Time Series Prediction.  _IEEE Transactions on Fuzzy Systems_ 29 (9), 2647–2660 (2021).  https://doi.org/10.1109/TFUZZ.2020.3005293 .
  * (188) Park, S., Lee, S. J., Weiss, E. & Motai, Y.  Intra- and Inter-Fractional Variation Prediction of Lung Tumors Using Fuzzy Deep Learning.  _IEEE Journal of Translational Engineering in Health and Medicine_ 4, 1–12 (2016).  https://doi.org/10.1109/JTEHM.2016.2516005 .
  * (189) Magdalena, L.  Semantic interpretability in hierarchical fuzzy systems: Creating semantically decouplable hierarchies.  _Information Sciences_ 496, 109–123 (2019).  https://doi.org/10.1016/j.ins.2019.05.016 .
  * (190) Wang, G., Jia, Q.-S., Qiao, J., Bi, J. & Liu, C.  A sparse deep belief network with efficient fuzzy learning framework.  _Neural Networks_ 121, 430–440 (2020).  https://doi.org/10.1016/j.neunet.2019.09.035 .
  * (191) Wang, G., Qiao, J., Bi, J., Li, W. & Zhou, M.  TL-GDBN: Growing Deep Belief Network With Transfer Learning.  _IEEE Transactions on Automation Science and Engineering_ 16 (2), 874–885 (2019).  https://doi.org/10.1109/TASE.2018.2865663 .
  * (192) Li, F., Qiao, J., Han, H. & Yang, C.  A self-organizing cascade neural network with random weights for nonlinear system modeling.  _Applied Soft Computing_ 42, 184–193 (2016).  https://doi.org/10.1016/j.asoc.2016.01.028 .
  * (193) Han, H.-G., Zhang, H.-J. & Qiao, J.-F.  Robust Deep Neural Network Using Fuzzy Denoising Autoencoder.  _International Journal of Fuzzy Systems_ 22 (4), 1356–1375 (2020).  https://doi.org/10.1007/s40815-020-00845-6 .
  * (194) George, S. & Santra, A. K.  Fuzzy Inspired Deep Belief Network for the Traffic Flow Prediction in Intelligent Transportation System Using Flow Strength Indicators.  _Big Data_ 8 (4), 291––307 (2020).  https://doi.org/10.1089/big.2019.0007 .
  * (195) Borkar, G. M. & Mahajan, A. R.  A secure and trust based on-demand multipath routing scheme for self-organized mobile ad-hoc networks.  _Wireless Networks_ 23 (8), 2455––2472 (2016).  https://doi.org/10.1007/s11276-016-1287-y .
  * (196) Chen, J., Yuan, W., Cao, J. & Lv, H.  Traffic-flow prediction via granular computing and stacked autoencoder.  _Granular Computing_ 5 (4), 449––459 (2019).  https://doi.org/10.1007/s41066-019-00167-5 .
  * (197) Liao, C.-W., Wang, I.-C., Lin, K.-P. & Lin, Y.-J.  A Fuzzy Seasonal Long Short-Term Memory Network for Wind Power Forecasting.  _Mathematics_ 9 (11) (2021).  https://doi.org/10.3390/math9111178 .
  * (198) Chang, P.-T.  Fuzzy seasonality forecasting.  _Fuzzy Sets and Systems_ 90 (1), 1–10 (1997).  https://doi.org/10.1016/S0165-0114(96)00138-8 .
  * (199) Imani, M. _et al._ Application of Rough and Fuzzy Set Theory for Prediction of Stochastic Wind Speed Data Using Long Short-Term Memory.  _Atmosphere_ 12 (7) (2021).  https://doi.org/10.3390/atmos12070924 .
  * (200) Pawlak, Z.  Rough sets.  _International Journal of Computer & Information Sciences_ 11 (5), 341––356 (1982).  https://doi.org/10.1007/bf01001956 .
  * (201) Liu, L., Liu, F. & Zheng, Y.  A Novel Ultra-Short-Term PV Power Forecasting Method Based on DBN-Based Takagi-Sugeno Fuzzy Model.  _Energies_ 14 (20) (2021).  https://doi.org/10.3390/en14206447 .
  * (202) Chimatapu, R., Hagras, H., Kern, M. & Owusu, G.  (ed.) _Hybrid Deep Learning Type-2 Fuzzy Logic Systems For Explainable AI_.  (ed.) _2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)_ , 1–6 (2020).
  * (203) S.N., D., S., G. & T.S., A.  (ed.) _Fuzzy Echo State Neural Network with Differential Evolution Framework for Time Series Forecasting_.  (ed.) _2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA)_ , 1322–1327 (2018).
  * (204) Mahmoud, T. A. & Elshenawy, L. M.  TSK fuzzy echo state neural network: a hybrid structure for black-box nonlinear systems identification.  _Neural Computing and Applications_ 1–19 (2022).  https://doi.org/10.1007/s00521-021-06838-2 .
  * (205) Wu, K., Liu, J., Liu, P. & Yang, S.  Time Series Prediction Using Sparse Autoencoder and High-Order Fuzzy Cognitive Maps.  _IEEE Transactions on Fuzzy Systems_ 28 (12), 3110–3121 (2020).  https://doi.org/10.1109/TFUZZ.2019.2956904 .
  * (206) Sevakula, R. K. & Verma, N. K.  (ed.) _Fuzzy Rule Reduction using Sparse Auto-Encoders_.  (ed.) _2015 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)_ , 1–7 (2015).
  * (207) Ojha, V., Abraham, A. & Snášel, V.  Heuristic design of fuzzy inference systems: A review of three decades of research.  _Engineering Applications of Artificial Intelligence_ 85, 845–864 (2019).  https://doi.org/10.1016/j.engappai.2019.08.010 .