Optimize TSK Fuzzy Systems for Regression Problems: Mini-Batch Gradient Descent with Regularization, DropRule and AdaBound (MBGD-RDA)
Dongrui Wu Ye Yuan Yihua Tan







Optimize TSK Fuzzy Systems for Regression Problems: Mini-Batch Gradient Descent with Regularization, DropRule, and AdaBound (MBGD-RDA)
    Dongrui Wu, Ye Yuan, Jian Huang, and Yihua Tan
D. Wu, Y. Yuan, J. Huang and Y. Tan are with the Ministry of Education Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China. Email: drwu@hust.edu.cn, yye@hust.edu.cn, huang_jan@hust.edu.cn, yhtan@hust.edu.cn.
D. Wu and Y. Tan are the corresponding authors.
    
========================================================================================================================================================================================================================================================================================================================================================================================================================================




Takagi-Sugeno-Kang (TSK) fuzzy systems are very useful machine learning models for regression problems. However, to our knowledge, there has not existed an efficient and effective training algorithm that ensures their generalization performance, and also enables them to deal with big data. Inspired by the connections between TSK fuzzy systems and neural networks, we extend three powerful neural network optimization techniques, i.e., mini-batch gradient descent (MBGD), regularization, and AdaBound, to TSK fuzzy systems, and also propose three novel techniques (DropRule, DropMF, and DropMembership) specifically for training TSK fuzzy systems. Our final algorithm, MBGD with regularization, DropRule and AdaBound (MBGD-RDA), can achieve fast convergence in training TSK fuzzy systems, and also superior generalization performance in testing. It can be used for training TSK fuzzy systems on datasets of any size; however, it is particularly useful for big datasets, on which currently no other efficient training algorithms exist.



Fuzzy systems, ANFIS, mini-batch gradient descent, regularization, AdaBound, DropRule






§ INTRODUCTION


Fuzzy systems <cit.>, particularly Takagi-Sugeno-Kang (TSK) fuzzy systems <cit.>, have achieved great success in numerous applications. This paper focuses on the applications of TSK fuzzy systems in machine learning <cit.>, particularly, supervised regression problems. In such problems, we have a training dataset with N labeled examples {𝐱_n,y_n}_n=1^N, where 𝐱_n∈ℝ^M×1, and would like to train a TSK fuzzy system to model the relationship between y and 𝐱, so that an accurate prediction can be made for any future unseen 𝐱.

There are generally three different strategies for optimizing a TSK fuzzy system in supervised regression[Some novel approaches for optimizing evolving fuzzy systems have also been proposed recently <cit.>; however, they are not the focus of this paper, so their details are not included.]:


  * Evolutionary algorithms <cit.>, in which each set of the parameters of the antecedent membership functions (MFs) and the consequents are encoded as an individual in a population, and genetic operators, such as selection, crossover, mutation, and reproduction, are used to produce the next generation. Generally, the overall fitness improves in each new generation, and a global optimum may be found given enough number of generations.

  * Gradient descent (GD) <cit.>, in which the parameters are moved in the negative gradient direction of the loss function to find its local minimum. Back-propagation  <cit.> is frequently used to calculate the gradients. These fuzzy systems are called neuro-fuzzy systems in the literature <cit.>.

  * GD and least squares estimation (LSE) <cit.>, which is used in the popular adaptive-network-based fuzzy inference system (ANFIS). The antecedent parameters are optimized by GD, and the consequent parameters by LSE. This approach usually converges much faster than using GD only.


However, all three strategies may have challenges in big data applications <cit.>. It's well-known that big data has at least three Vs[There may be also other Vs, e.g., veracity, value, etc.] <cit.>: volume (the size of the data), velocity (the speed of the data), and variety (the types of data). Volume means that the number of training examples (N) is very large, and/or the dimensionality of the input (M) is very high. Fuzzy systems, and actually almost all machine learning models, suffer from the curse of dimensionality, i.e., the number of rules (parameters) increases exponentially with M. However, in this paper we assume that the dimensionality can be reduced effectively to just a few, e.g., using principal component analysis <cit.>. We mainly consider how to deal with large N.

Evolutionary algorithms are not suitable for optimizing TSK fuzzy systems when N is large, because they have very high memory and computing power requirement: they need to evaluate the fitness of each individual on the entire training dataset (which may be too large to be loaded into the memory completely), and there are usually tens or hundreds of individuals in a population, and tens or hundreds of generations are needed to find a good solution. ANFIS may result in significant overfitting in regression problems, as demonstrated in Section <ref> of this paper. So, we focus on GD.

When N is small, batch GD can be used to compute the average gradients over all N training examples, and then update the model parameters. When N is large, there may not be enough memory to load the entire training dataset, and hence batch GD may be very slow or even impossible to perform. In such cases, stochastic GD can be used to compute the gradients for each training example, and then update the model parameters. However, the stochastic gradients may have very large variance, and hence the training may be unstable. A good compromise between batch GD and stochastic GD, which has achieved great success in deep learning <cit.>, is mini-batch gradient descent (MBGD). It randomly selects a small number (typically 32 or 64 <cit.>) of training examples to compute the gradients and update the model parameters. MBGD is a generic approach not specific to a particular model to be optimized, so it should also be applicable to the training of fuzzy systems. In fact, <cit.> has compared the performances of full-batch GD, MBGD and stochastic GD on the training of Mamdani neuro-fuzzy systems, and showed that MBGD achieved the best performance. This paper applies MBGD to the training of TSK fuzzy systems.

In MBGD, the learning rate is very important to the convergence speed and quality in training. Many different schemes, e.g., momentum <cit.>, averaging <cit.>, AdaGrad <cit.>, RMSProp <cit.>, Adam <cit.>, etc., have been proposed to optimize the learning rate in neural network training. Adam may be the most popular one among them. However, to the knowledge of the authors, only a short conference paper <cit.> has applied Adam to the training of simple single-input rule modules fuzzy systems <cit.>. Very recently, an improvement to Adam, AdaBound <cit.>, was proposed, which demonstrated faster convergence and better generalization than Adam. To our knowledge, no one has used AdaBound for training TSK fuzzy systems.

In addition to fast convergence, the generalization ability of a machine learning model is also crucially important. Generalization means the model must perform well on previously unobserved inputs (not just the known training examples).

Regularization is frequently used to reduce overfitting and improve generalization. According to Goodfellow et al. <cit.>, regularization is “any modification we make to a learning algorithm that is intended to reduce its generalization error but not its training error." It has also been used in training TSK fuzzy systems to increase their performance and interpretability <cit.>. For example, Johansen <cit.>, and  Lughofer and Kindermann <cit.>, used ℓ_2 regularization (also known as weight decay, ridge regression, or Tikhonov regularization) to stabilize the matrix inversion operation in LSE. Jin <cit.> used regularization to merge similar MFs into a single one to reduce the size of the rulebase and hence to increase the interpretability of the fuzzy system. Lughofer and Kindermann <cit.>, and Luo et al. <cit.>, used sparsity regularization to identify a TSK fuzzy system with a minimal number of fuzzy rules and a minimal number of non-zero consequent parameters. All these approaches used LSE to optimize the TSK rule consequents, which may result in significant overfitting in regression problems (Section <ref>). To our knowledge, no one has integrated MBGD and regularization for TSK fuzzy system training.

Additionally, some unique approaches have also been proposed in the last few years to reduce overfitting and increase generalization of neural networks, particularly deep neural networks, e.g., DropOut <cit.> and DropConnect <cit.>. DropOut randomly discards some neurons and their connections during the training, which prevents units from co-adapting too much. DropConnect randomly sets some connection weights to zero during the training. Although DropOut and DropConnect have demonstrated outstanding performance and hence widely used in deep learning, no similar techniques exist for training TSK fuzzy systems.

This paper fills the gap in efficient and effective training of TSK fuzzy systems, particularly for big data regression problems. Its main contributions are:


  * Inspired by the connections between TSK fuzzy systems and neural networks <cit.>, we extend three powerful neural network optimization techniques, i.e., MBGD, regularization, and AdaBound, to TSK fuzzy systems.

  * We propose three novel techniques (DropRule, DropMF, and DropMembership) specifically for training TSK fuzzy systems.

  * Our final algorithm, MBGD with regularization, DropRule and AdaBound (MBGD-RDA), demonstrates superior performance on 10 real-world datasets from various application domains, of different sizes.


The remainder of this paper is organized as follows: Section <ref> introduces our proposed MBGD-RDA algorithm. Section <ref> presents our experimental results. Section <ref> draws conclusion and points out some future research directions.




§ THE MBGD-RDA ALGORITHM
 

This section introduces our proposed MBGD-RDA algorithm for training TSK fuzzy systems, whose pseudo-code is given in Algorithm <ref> and Matlab implementation at https://github.com/drwuHUST/MBGD_RDA. Note that θ_K returned from Algorithm <ref> is not necessarily the optimal one among {θ_k}_k=1^K, i.e., the one that gives the smallest test error. The iteration number corresponding to the optimal θ^* can be estimated using early stopping <cit.>. However, this is beyond the scope of this paper. Herein, we assume that the user has pre-determined K.

[htbp] 
N labeled training examples {𝐱_n,y_n}_n=1^N, where 𝐱_n=(x_n,1,...,x_n,M)^T∈ℝ^M×1 
 L(θ), the loss function for the TSK fuzzy system parameter vector θ 
 M_m, the number of Gaussian MFs in the mth input domain 
 K, the maximum number of training iterations 
 N_bs∈[1,N], the mini-batch size 
 P∈(0,1), the DropRule rate 
 α, the initial learning rate (step size) 
 λ, the ℓ_2 regularization coefficient 
 β_1,β_2∈[0,1), exponential decay rates for the moment estimates in AdaBound 
 ϵ, a small positive number in AdaBound 
 l(k) and u(k), the lower and upper bound functions in AdaBound 
The final θ.

m=1,...,M
Compute the minimum and maximum of all {x_n,m}_n=1^N 
Initialize the center of the M_m Gaussian MFs uniformly between the minimum and the maximum 
Initialize the standard deviation of all M_m Gaussian MFs as the standard deviation of {x_n,m}_n=1^N 
Initialize the consequent parameters of all R rules as 0 
θ_0 is the concatenation of all Gaussian MF centers, standard deviations, and rule consequent parameters 

𝐦_0=0;
𝐯_0=0 
k=1,...,K

Randomly select N_bs training examples 
n=1,...,N_bs
r=1,...,R
DropRule
f_r(𝐱_n)=0 
Generate p, a uniformly distributed random number in [0,1] 
p≤ P
Compute f_r(𝐱_n), the firing level of 𝐱_n on Rule_r 
Compute y(𝐱_n), the TSK fuzzy system output for 𝐱_n, by (<ref>) 
Compute the gradients
each element θ_k-1(i) in θ_k-1

    𝐠_k(i)={[   ∂ L/∂θ_k-1(i), θ_k-1(i)  y(𝐱_n);               0,                  ].

ℓ_2 regularization
Identify the index set I, which consists of the elements of θ corresponding to the rule consequent coefficients, excluding the bias terms 
each index i∈ I
𝐠_k(i)=𝐠_k(i)+λ·θ_k-1(i) 
AdaBound
𝐦_k=β_1m_k-1+(1-β_1)g_k;  v_k=β_2v_k-1+(1-β_2)g_k^2 
m̂_k=m_k/1-β_1^k;   v̂_k=v_k/1-β_2^k 
α̂=max[l(k),min(u(k),
α/√(v̂_t)+ϵ)] 
θ_k=θ_k-1-α̂⊙m̂_k 

Return θ_K
The MBGD-RDA algorithm for TSK fuzzy system optimization. Typical values of some hyper-parameters are: β_1=0.9, β_2=0.999, and ϵ=10^-8. 


The key notations used in this paper are summarized in Table <ref>. The details of MBGD-RDA are explained next.







 §.§ The TSK Fuzzy System


Assume the input 𝐱=(x_1,...,x_M)^T∈ℝ^M×1, and the TSK fuzzy system has R rules:

    Rule_r:     x_1  X_r,1⋯ x_M  X_r,M, 
        y_r(𝐱)=b_r,0+∑_m=1^M b_r,mx_m,

where X_r,m (r=1,...,R; m=1,...,M) are fuzzy sets, and b_r,0 and b_r,m are consequent parameters. This paper considers only Gaussian MFs, because their derivatives are easier to compute <cit.>. However, our algorithm can also be applied to other MF shapes, as long as their derivatives can be computed.

The membership grade of x_m on a Gaussian MF X_r,m is:

    μ_X_r,m(x_m)=exp(-(x_m-c_r,m)^2/2σ_r,m^2),

where c_r,m is the center of the Gaussian MF, and σ_r,m the standard deviation.

The firing level of Rule_r is:

    f_r(𝐱)=∏_m=1^Mμ_X_r,m(x_m),

and the output of the TSK fuzzy system is:

    y(𝐱)=∑_r=1^R f_r(𝐱)y_r(𝐱)/∑_r=1^R f_r(𝐱).

Or, if we define the normalized firing levels as:

    f̅_r(𝐱)=f_r(𝐱)/∑_k=1^Rf_k(𝐱),   r=1,...,R

then, (<ref>) can be rewritten as:

    y(𝐱)   =∑_r=1^Rf̅_r(𝐱)· y_r(𝐱).


To optimize the TSK fuzzy system, we need to tune c_r,m, σ_r,m, b_r,0 and b_r,m, where r=1,...,R and m=1,...,M.



 §.§ Regularization


Assume there are N training examples {𝐱_n,y_n}_n=1^N, where 𝐱_n=(x_n,1,...,x_n,M)^T∈ℝ^M×1.

In this paper, we use the following ℓ_2 regularized loss function:

    L=1/2∑_n=1^N_bs[y_n-y(𝐱_n)]^2+λ/2∑_r=1^R∑_m=1^M b_r,m^2,

where N_bs∈[1,N], and λ≥ 0 is a regularization parameter. Note that b_r,0 (r=1,...,R) are not regularized in (<ref>). As pointed out by Goodfellow et al. <cit.>, for neural networks, one typically penalizes only the weights of the affine transformation at each layer and leaves the biases un-regularized. The biases typically require less data to fit accurately than the weights. The biases in neural networks are corresponding to the b_r,0 terms here, so we take this typical approach, and leave b_r,0 un-regularized.




 §.§ Mini-Batch Gradient Descent (MBGD)


The gradients of the loss function (<ref>) are given in (<ref>)-(<ref>), where Φ(r,m) is the index set of the rules that contain X_r,m, x_n,0≡1, and I(m) is an indicator function:

    I(m)={[  0, m=0;  1, m>0 ].

I(m) ensures that b_r,0 (r=1,...,R) are not regularized.



In MBGD, each time we randomly sample N_bs∈[1, N] training examples, compute the gradients from them, and then update the antecedent and consequent parameters of the TSK fuzzy system. Let θ_k be the model parameter vector in the kth iteration, and ∂L/∂θ_k be the first-order gradients computed from  (<ref>)-(<ref>). Then, the update rule is:

    θ_k=θ_k-1-α∂L/∂θ_k-1,

where α>0 is the learning rate (step size).

When N_bs=1, MBGD degrades to the stochastic GD. When N_bs=N, it becomes the batch GD.



 §.§ DropRule


DropRule is a novel technique to reduce overfitting and increase generalization in training TSK fuzzy systems, inspired by the well-known DropOut <cit.> and DropConnect <cit.> techniques in deep learning. DropOut randomly discards some neurons and their connections during the training. DropConnect randomly sets some connection weights to zero during the training. DropRule randomly discards some rules during the training, but uses all rules in testing.

Let the DropRule rate be P∈(0,1). For each training example in the iteration, we set the firing level of a rule to its true firing level with probability P, and to zero with probability 1-P. The output of the TSK fuzzy system is again computed by a firing level weighted average of the rule consequents. Since the firing levels of certain rules are artificially set to zero, they do not contribute anything to the fuzzy system output, i.e., they are artificially dropped for this particular training example, as shown in Fig. <ref>[The ANFIS representation of a TSK fuzzy system is used here. For details, please refer to Section <ref>.]. Then, GD is applied to update the model parameters in rules that are not dropped (the parameters in the dropped rules are not updated for this particular training example).



When the training is done, all rules will be used in computing the output for a new input, just as in a traditional TSK fuzzy system. This is different from DropOut and DropConnect for neural networks, which need some special operations in testing to ensure the output does not have a bias. We do not need to pay special attention in using a TSK fuzzy system trained from DropRule, because the final step of a TSK fuzzy system is a weighted average, which removes the bias automatically.

The rationale behind DropOut is that <cit.> “each hidden unit in a neural network trained with dropout must learn to work with a randomly chosen sample of other units. This should make each hidden unit more robust and drive it towards creating useful features on its own without relying on other hidden units to correct its mistakes." This is also the motivation of DropRule: by randomly dropping some rules, we force each rule to work with a randomly chosen subset of rules, and hence each rule should maximize its own modeling capability, instead of relying too much on other rules. This may help increase the generalization of the final TSK fuzzy system. Our experiments in the next section demonstrate that DropRule alone may not always offer advantages, but it works well when integrated with an efficient learning rate adaptation algorithm like AdaBound.



 §.§ Adam and AdaBound


As mentioned in the Introduction, the learning rate is a very important hyper-parameter in neural network training, which is also the case for TSK fuzzy system training. Among the various proposals for adjusting the learning rate, Adam <cit.> may be the most popular one. It computes an individualized adaptive learning rate for each different model parameter from the estimates of the first and second moments of the gradient. Essentially, it combines the advantages of two other approaches: AdaGrad <cit.>, which works well with sparse gradients, and RMSProp <cit.>, which works well in online and non-stationary settings.

Very recently, an improvement to Adam, AdaBound <cit.>, was proposed. It bounds the individualized adaptive learning rate from the upper and the lower, so that an extremely large or small learning rate cannot occur. Additionally, the bounds become tighter as the number of iterations increases, which forces the learning rate to approach a constant (as in the stochastic GD). AdaBound has demonstrated faster convergence and better generalization than Adam in <cit.>, so it is adopted in this paper.

The pseudo-code of AdaBound can be found in Algorithm <ref>. By substituting L in (<ref>) into it, we can use the bounded individualized adaptive learning rates for different elements of θ, which may result in better training and generalization performance than using a fixed learning rate. The lower and upper bound functions used in this paper were similar to those in <cit.>:

    l(k)   =0.01-0.01/(1-β_2)k+1
    
    u(k)   =0.01+0.01/(1-β_2)k

When k=0, the bound is [0,+∞). When k approaches +∞, the bound approaches [0.01,0.01].




§ EXPERIMENTS
 

This section presents experimental results to demonstrate the performance of our proposed MBGD-RDA.



 §.§ Datasets


Ten regression datasets from the CMU StatLib Datasets Archive[http://lib.stat.cmu.edu/datasets/] and the UCI Machine Learning Repository[http://archive.ics.uci.edu/ml/index.php] were used in our experiments. Their summary is given in Table <ref>. Their sizes range from small to large.



Nine of the 10 datasets have only numerical features. Abalone has a categorical feature (sex: male, female, and infant), which was ignored in our experiments[We also tried to convert the categorical feature into numerical ones using one-hot coding and use them together with the other seven numerical features. However, the RMSEs were worse than simply ignoring it.]. Each numerical feature was z-normalized to have zero mean and unit variance, and the output mean was also subtracted. Because fuzzy systems have difficulty dealing with high dimensional data, we constrained the maximum input dimensionality to be five: if a dataset had more than five features, then principal component analysis was used to reduce them to five.

The TSK fuzzy systems had M_m=2 Gaussian MFs in each input domain. For M inputs, the TSK fuzzy system has 2MM_m+(M+1)·(M_m)^M parameters.



 §.§ Algorithms


We compared the performances of the following seven algorithms[We also tried to use support vector regression <cit.> as a baseline regression model; however, it was too time-consuming to train on big datasets. So, we abandoned it.]:


  * Ridge regression <cit.>, with ridge coefficient λ=0.05. This algorithm is denoted as  in the sequel.


  * MBGD, which is a mini-batch version of the batch GD algorithm introduced in <cit.>. The batch size N_bs was 64, the initial learning rate α was 0.01, and the adaptive learning rate adjustment rule in <cit.> was implemented: α was multiplied by 1.1 if the loss function was reduced in four successive iterations, and by 0.9 if the loss function had two consecutive combinations of an increase followed by a decrease. This algorithm is denoted as  in the sequel.


  * MBGD with Regularization, which was essentially identical to , except that the loss function had an ℓ_2 regularization term on the consequent parameters [see (<ref>)]. λ=0.05 was used. This algorithm is denoted as  in the sequel.


  * MBGD with DropRule, which was essentially identical to , except that DropRule with P=0.5 was also used in the training, i.e., for each training example, we randomly set the firing level of 50% rules to zero. This algorithm is denoted as  in the sequel.


  * MBGD with Regularization and DropRule, which integrated  and . This algorithm is denoted as  in the sequel.


  * MBGD with AdaBound, which was essentially identical to , except that the learning rates were adjusted by AdaBound. α=0.01, β_1=0.9, β_2=0.999, and ϵ=10^-8 were used. This algorithm is denoted as  in the sequel.


  * MBGD with Regularization, DropRule and AdaBound, which combined ,  and . This algorithm is denoted as  in the sequel.

For clarity, the parameters of these seven algorithms are also summarized in Table <ref>.



For each dataset, we randomly selected 70% examples for training, and the remaining 30% for test.  was trained in one single pass on all training examples, and then its root mean squared error (RMSE) on the test examples was computed. The other six MBGD-based algorithms were iterative. The TSK fuzzy systems had two Gaussian MFs in each input domain. Their centers were initialized at the minimum and maximum of the input domain, and their standard deviations were initialized to the standard deviation of the corresponding input. All rule consequent parameters were initialized to zero. The maximum number of iterations was 500. After each training iteration, we recorded the corresponding test RMSE of each algorithm. Because there was randomness involved (e.g., the training/test data partition, the selection of the mini-batches, etc.), each algorithm was repeated 10 times on each dataset, and the average test results are reported next.



 §.§ Experimental Results


The average test RMSEs of the seven algorithms are shown in Fig. <ref>. We can observe that:


  * ,  and  had comparable performance with . All of them were worse than the simple  on seven out of the 10 datasets, suggesting that a model with much more parameters and nonlinearity does not necessarily outperform a simple linear regression model, if not properly trained.


  *  and  performed the best among the seven algorithms. On nine out of the 10 datasets (except Wine-Red), 's best test RMSEs were smaller than . On all 10 datasets, 's best test RMSEs were smaller than .  and  also converged much faster than , ,  and . As the final TSK fuzzy systems trained from the six MBGD-based algorithms had the same structure and the same number of parameters, these results suggest that AdaBound was indeed very effective in updating the learning rates, which in turn helped obtain better learning performances.


  * Although regularization alone (), DropRule alone (), and the combination of regularization and DropRule (), did not result in much performance improvement (i.e., , ,  and   had similar performances),  outperformed  on three out of the 10 datasets, and they had comparable performances on many other datasets. These results suggest that using an effective learning rate updating scheme like AdaBound can help unleash the power of regularization and DropRule, and hence achieve better learning performance.




To better visualize the performance differences among the six MBGD-based algorithms, we plot in Fig. <ref> the percentage improvements of , , ,  and  over : in each iteration, we treat the test RMSE of  as one, and compute the relative percentage improvements of the test RMSEs of the other five MBGD-based algorithms over it. For example, let RMSE_GD(k) and RMSE_MBGD-RDA(k) be the test RMSEs of  and  at iteration k, respectively. Then, the percentage improvement of the test RMSE of  over  at iteration k is:

    p(k)=100×RMSE_GD(k)-RMSE_MBGD-RDA(k)/RMSE_GD(k).


Fig. <ref> confirmed the observations made from Fig. <ref>. Particularly,  and  converged much faster and to smaller values than , ,  and ; the best test RMSEs of  and  were also much smaller than those of , ,  and . Among the five enhancements to , only  consistently outperformed . In other words, although  may not always outperform , its performance was more stable than ; so, it should be preferred over  in practice.



The time taken to finish 500 training iterations for the MBGD-based algorithms on the 10 datasets is shown in Table <ref>. The platform was a desktop computer running Matlab 2018a and Windows 10 Enterprise 64x, with Intel Core i7-8700K CPU @ 3.70 GHz, 16GB memory, and 512GB solid state drive. The CPU has 12 cores, but each algorithm used only one core. Not surprisingly,  was much faster than others, because it has a closed-form solution, and no iteration was needed. Among the six MBGD-based algorithms,  was the fastest. One reason is that DropRule reduced the number of parameters to be adjusted in each iteration.





 §.§ Parameter Sensitivity


It's also important to study the sensitivity of  to its hyper-parameters. Algorithm <ref> shows that  has the following hyper-parameters:


  * M_m, the number of Gaussian MFs in the mth input domain

  * K, the maximum number of training iterations

  * N_bs∈[1,N], the mini-batch size

  * P∈(0,1), the DropRule rate

  * α, the initial learning rate (step size)

  * λ, the ℓ_2 regularization coefficient

  * β_1,β_2∈[0,1), exponential decay rates for the moment estimates in AdaBound

  * ϵ, a small positive number in AdaBound

  * l(k) and u(k), the lower and upper bound functions in AdaBound

Among them, M_m is a parameter for all TSK fuzzy systems, not specific to ; K can be determined by early-stopping on a validation dataset; and, β_1, β_2, ϵ, l(k) and u(k) are AdaBound parameters, whose default values are usually used. So, we only studied the sensitivity of  to N_bs, P, α and λ, which are unique to .

The results, in terms of the test RMSEs, on the PM10 dataset are shown in Fig. <ref>, where each experiment was repeated 100 times and the average test RMSEs are shown. In each subfigure, except for the hyper-parameter under consideration, the values for other parameters were: M_m=2, K=500, N_bs=64, P=0.5, α=0.01, λ=0.05, β_1=0.9, β_2=0.999, ϵ=10^-8, and l(k) and u(k) are defined in (<ref>) and (<ref>), respectively. Clearly,  is stable with respect to each of the four hyper-parameters in a wide range, which is desirable.







 §.§ Comparison with ANFIS
 

ANFIS <cit.> is an efficient algorithm for training TSK fuzzy systems on small datasets. This subsection compares the performance of  with ANFIS on the first six smaller datasets.

The ANFIS structure of a two-input one-output TSK fuzzy system is shown in Fig. <ref>. It has five layers:


  * Layer 1: The membership grade of x_m on X_r,m is computed, by (<ref>).

  * Layer 2: The firing level of each rule Rule_r is computed, by (<ref>).

  * Layer 3: The normalized firing levels of the rules are computed, by (<ref>).

  * Layer 4: Each normalized firing level is multiplied by its corresponding rule consequent.

  * Layer 5: The output is computed, by (<ref>).

All parameters of the ANFIS, i.e., the shapes of the MFs and the rule consequents, can be trained by GD <cit.>. Or, to speed up the training, the antecedent parameters can be tuned by GD, and the consequent parameters by LSE <cit.>.



In the experiment, we used the anfis function in Matlab 2018b, which does not allow us to specify a batch size, but to use all available training examples in each iteration. For fair comparison, in  we also set the batch size to the number of training examples. anfis in Matlab has two optimization options: 1) batch GD for both antecedent and consequent parameters (denoted as  in the sequel); and, 2) batch GD for antecedent parameters, and LSE for consequent parameters (denoted as  in the sequel). We compared  with both options.

The training and test RMSEs, averaged over 10 runs, are shown in Fig. <ref>.  always converged much faster than , and its best test RMSE was also always smaller. Additionally, it should be emphasized that  can be used also for big data, whereas  cannot.




Interestingly, although  always achieved the smallest training RMSE, its test RMSE was almost always the largest, and had large oscillations. This suggests that  had significant overfitting. If we could reduce this overfitting, e.g., through regularization, then  could be a very effective TSK fuzzy system training algorithm for small datasets. This is one of our future research directions.

Fig. <ref> shows the learning rates of ,  and . For the first two ANFIS based approaches, all model parameters shared the same learning rate. However, in , different model parameters had different learning rates, and we show the average learning rate across all model parameters on each dataset. The learning rates in  and  first gradually increased and then decreased. Interestingly, the learning rate of  was almost always smaller than that of  when the number of iterations was large. The learning rate of  was always very high at the beginning, and then rapidly decreased. The initial high learning rate helped  achieve rapid convergence.



 §.§ Comparison with DropMF and DropMembership


In addition to DropRule, there could be other DropOut approaches in training a TSK fuzzy system, e.g.,


  * DropMF, in which each input MF is dropped with a probability 1-P, as illustrated in Fig. <ref>. Dropping an MF is equivalent to setting the firing level of that MF to 1 (instead of 0). Comparing DropMF in Fig. <ref> and DropRule in Fig. <ref>, we can observe that each DropRule operation reduces the number of used rules by one; on the contrary, DropMF does not reduce the number of used rules; instead, it reduces the number of antecedents in multiple rules by one.




  * DropMembership, in which the membership of an input in each MF is dropped with a probability 1-P, as illustrated in Fig. <ref>. Dropping a membership is equivalent to setting that membership to 1 (instead of 0). Comparing DropMembership in Fig. <ref> and DropMF in Fig. <ref>, we can observe that DropMembership has a smaller impact on the firing levels of the rules than DropMF. For example, in Fig. <ref>, both f_1 and f_2 are impacted by DropMF, whereas in Fig. <ref>, only f_1 is impacted by DropMembership.





Next, we compare the performances of DropMF, DropMembership with DropRule, by replacing DropRule in  by DropMF and DropMembership, respectively. The training and test RMSEs, averaged over 10 runs, are shown in Fig. <ref>. Generally, DropRule performed the best, and DropMembership the worst. Comparing Figs. <ref>, <ref> and <ref>, we can observe that DropRule introduces the maximum change to the TSK fuzzy system structure, and DropMembership the smallest. This suggests that a dropout operation that introduces more changes to the TSK fuzzy system may be more beneficial to the training and test performances.





 §.§ Comparison with Adam


We also compared the performances of AdaBound with Adam. The learning algorithm, , was identical to , except that AdaBound was replaced by Adam, by setting l(k)=0 and u(k)=+∞ in Algorithm <ref>.



The training and test RMSEs, averaged over 10 runs, are shown in Fig. <ref>.  converged faster than, or equally fast with, , and had smaller or comparable best test RMSEs as  on most datasets. So, it is generally safe to choose AdaBound over Adam.



§ CONCLUSION AND FUTURE RESEARCH
 

TSK fuzzy systems are very useful machine learning models for regression problems. However, to our knowledge, there has not existed an efficient and effective training algorithm that enables them to deal with big data. Inspired by the connections between TSK fuzzy systems and neural networks, this paper extended three powerful optimization techniques for neural networks, e.g., MBGD, regularization, and AdaBound, to TSK fuzzy systems, and also proposed three novel techniques (DropRule, DropMF, and DropMembership) specifically for training TSK fuzzy systems. Our final algorithm, MBGD-RDA, which integrates MBGD, regularization, AdaBound and DropRule, can achieve fast convergence in training TSK fuzzy systems, and also superior generalization performance in testing. It can be used for training TSK fuzzy systems on datasets of any size; however, it is particularly useful for big datasets, for which currently no other efficient training algorithms exist. We expect that our algorithm will help promote the applications of TSK fuzzy systems, particularly to big data.

Finally, we need to point out that we have not considered various uncertainties in the data, e.g., missing values, wrong values, noise, outliers, etc., which frequently happen in real-world applications. Some techniques, e.g., rough sets <cit.>, could be integrated with fuzzy sets to deal with them. Or, the type-1 TSK fuzzy systems used in this paper could also be extended to interval or general type-2 fuzzy systems <cit.> to cope with more uncertainties. These are some of our future research directions.



10

url@samestyle








Mendel2017
J. M. Mendel, Uncertain rule-based fuzzy systems: introduction and new
  directions, 2nd ed.1em plus 0.5em minus 0.4emSpringer, 2017.

Nguyen2019
A. Nguyen, T. Taniguchi, L. Eciolaza, V. Campos, R. Palhares, and
  M. Sugeno, “Fuzzy control systems: Past, present and future,” IEEE
  Computational Intelligence Magazine, vol. 14, no. 1, pp. 56–68, 2019.

Couso2019
I. Couso, C. Borgelt, E. Hullermeier, and R. Kruse, “Fuzzy sets in
  data analysis: From statistical foundations to machine learning,” IEEE
  Computational Intelligence Magazine, vol. 14, no. 1, pp. 31–44, 2019.

Gu2019
X. Gu, P. Angelov, and H.-J. Rong, “Local optimality of self-organising
  neuro-fuzzy inference systems,” Information Sciences, vol. 503, pp.
  351–380, 2019.

Gu2019a
X. Gu and P. Angelov, “Self-boosting first-order autonomous learning
  neuro-fuzzy systems,” Applied Soft Computing, vol. 77, pp. 118–134,
  2019.

drwuEAAI2006
D. Wu and W. W. Tan, “Genetic learning and performance evaluation of type-2
  fuzzy logic controllers,” Engineering Applications of Artificial
  Intelligence, vol. 19, no. 8, pp. 829–841, 2006.

Wang1992b
L.-X. Wang and J. M. Mendel, “Back-propagation of fuzzy systems as nonlinear
  dynamic system identifiers,” in Proc. IEEE Int'l Conf. on Fuzzy
  Systems, San Diego, CA, 1992, pp. 1409–1418.

Rumelhart1986
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations
  by back-propagating errors.” Nature, vol. 323, pp. 533–536, 1986.

Lin1996a
C.-T. Lin and C. S. G. Lee, Neural Fuzzy Systems: a Neuro-Fuzzy Synergism
  to Intelligent Systems.1em plus 0.5em minus 0.4emUpper Saddle
  River, NJ: Prentice Hall, 1996.

Jang1993
J. R. Jang, “ANFIS: adaptive-network-based fuzzy inference system,”
  IEEE Trans. on Systems, Man, and Cybernetics, vol. 23, no. 3, pp.
  665–685, 1993.

Chen2014a
M. Chen, S. Mao, and Y. Liu, “Big data: A survey,” Mobile Networks and
  Applications, vol. 19, no. 2, pp. 171–209, 2014.

Wu2014
X. Wu, X. Zhu, G.-Q. Wu, and W. Ding, “Data mining with big data,” IEEE
  Trans. on Knowledge and Data Engineering, vol. 26, no. 1, pp. 97–107, 2014.

McAfee2012
A. McAfee, E. Brynjolfsson, T. H. Davenport, D. Patil, and D. Barton, “Big
  data: the management revolution,” Harvard Business Review, vol. 90,
  no. 10, pp. 60–68, 2012.

Jolliffe2002
I. Jolliffe, Principal Component Analysis.1em plus 0.5em minus
  0.4emWiley Online Library, 2002.

Goodfellow2016
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning.1em
  plus 0.5em minus 0.4emBoston, MA: MIT Press, 2016,
  <http://www.deeplearningbook.org>.

Bengio2012
Y. Bengio, “Practical recommendations for gradient-based training of deep
  architectures,” in Neural Networks: Tricks of the Trade, 2012, pp.
  437–478.

Nakasima-Lopez2019
S. Nakasima-López, J. R. Castro, M. A. Sanchez, O. Mendoza, and
  A. Rodríguez-Díaz, “An approach on the implementation of full batch,
  online and mini-batch learning on a Mamdani based neuro-fuzzy system with
  center-of-sets defuzzification: Analysis and evaluation about its
  functionality, performance, and behavior,” PLOS ONE, vol. 14,
  no. 9, pp. 1–40, 2019.

Polyak1992
B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approximation by
  averaging,” SIAM Journal on Control and Optimization, vol. 30,
  no. 4, pp. 838–855, 1992.

Duchi2011
J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online
  learning and stochastic optimization,” Journal of Machine Learning
  Research, vol. 12, pp. 2121–2159, 2011.

Tieleman2012
T. Tieleman and G. Hinton, “Lecture 6.5–RMSProp: Divide the gradient by a
  running average of its recent magnitude,” COURSERA: Neural Networks
  for Machine Learning, vol. 4, no. 2, pp. 26–31, 2012.

Kingma2015
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in
  Proc. 3rd Int'l Conf. on Learning Representations, San Diego, CA, May
  2015.

Matsumura2017
S. Matsumura and T. Nakashima, “Incremental learning for SIRMs fuzzy
  systems by Adam method,” in Joint 17th World Congress of Int‘l
  Fuzzy Systems Association and 9th Int’l Conf. on Soft Computing and
  Intelligent Systems, Otsu, Japan, June 2017, pp. 1–4.

Yubazaki1997
N. Yubazaki, J. Yi, M. Otani, and K. Hirota, “SIRMs dynamically connected
  fuzzy inference model and its applications,” in Proc. 7th IFSA World
  Congress, Prague, Czech Republic, Jun. 1997, pp. 410–415.

Luo2019
L. Luo, Y. Xiong, Y. Liu, and X. Sun, “Adaptive gradient methods with dynamic
  bound of learning rate,” arXiv preprint arXiv:1902.09843, 2019.

Johansen1996
T. A. Johansen, “Robust identification of Takagi-Sugeno-Kang fuzzy models
  using regularization,” in Proc. 5th IEEE Int'l Conf. on Fuzzy
  Systems, New Orleans, LA, Sep. 1996, pp. 180–193.

Jin2000
Y. Jin, “Fuzzy modeling of high-dimensional systems: complexity reduction and
  interpretability improvement,” IEEE Trans. on Fuzzy Systems, vol. 8,
  no. 2, pp. 212–221, 2000.

Lughofer2010
E. Lughofer and S. Kindermann, “SparseFIS: Data-driven learning of fuzzy
  systems with sparsity constraints,” IEEE Trans. on Fuzzy Systems,
  vol. 18, no. 2, pp. 396–411, 2010.

Luo2015
M. Luo, F. Sun, and H. Liu, “Dynamic T-S fuzzy systems identification based
  on sparse regularization,” Asian Journal of Control, vol. 17, no. 1,
  pp. 274–283, 2015.

Lughofer2008
E. Lughofer and S. Kindermann, “Improving the robustness of data-driven fuzzy
  systems with regularization,” in Proc. IEEE Int'l Conf. on Fuzzy
  Systems, Hong Kong, China, Jul. 2008, pp. 703–709.

Srivastava2014
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov,
  “Dropout: A simple way to prevent neural networks from overfitting,”
  Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958,
  2014.

Wan2013
L. Wan, M. Zeiler, S. Zhang, Y. LeCun, and R. Fergus, “Regularization of
  neural networks using DropConnect,” in Proc. Int'l Conf. on Machine
  Learning, Atlanta, GA, June 2013, pp. 1058–1066.

drwuTSK2019
D. Wu, C.-T. Lin, J. Huang, and Z. Zeng, “On the functional equivalence of
  TSK fuzzy systems to neural networks, mixture of experts, CART, and
  stacking ensemble regression,” IEEE Trans. on Fuzzy Systems, 2019,
  accepted.

drwuMFs2012
D. Wu, “Twelve considerations in choosing between Gaussian and trapezoidal
  membership functions in interval type-2 fuzzy logic controllers,” in
  Proc. IEEE World Congress on Computational Intelligence, Brisbane,
  Australia, June 2012.

Smola2004
A. J. Smola and B. Schölkopf, “A tutorial on support vector regression,”
  Statistics and Computing, vol. 14, no. 3, pp. 199–222, 2004.

Hoerl1970
A. E. Hoerl and R. W. Kennard, “Ridge regression: Biased estimation for
  nonorthogonal problems,” Technometrics, vol. 12, no. 1, pp. 55–67,
  1970.

Pawlak2012
Z. Pawlak, Rough sets: Theoretical aspects of reasoning about
  data.1em plus 0.5em minus 0.4emSpringer Science & Business
  Media, 2012, vol. 9.

drwuFundamental2012
D. Wu, “On the fundamental differences between interval type-2 and type-1
  fuzzy logic controllers,” IEEE Trans. on Fuzzy Systems, vol. 20,
  no. 5, pp. 832–848, 2012.




