Interpretable Neural Networks with Random Constructive Algorithm
This paper introduces an Interpretable Neural Network (INN) incorporating
spatial information to tackle the opaque parameterization process of random
weighted neural networks. The INN leverages spatial information to elucidate
the connection between parameters and network residuals. Furthermore, it
devises a geometric relationship strategy using a pool of candidate nodes and
established relationships to select node parameters conducive to network
convergence. Additionally, a lightweight version of INN tailored for
large-scale data modeling tasks is proposed. The paper also showcases the
infinite approximation property of INN. Experimental findings on various
benchmark datasets and real-world industrial cases demonstrate INN's
superiority over other neural networks of the same type in terms of modeling
speed, accuracy, and network structure.
Jing Nan, Wei Dai
An Interpretable Constructive Algorithm for Incremental Random Weight Neural Networks and Its Application

###### Abstract

Incremental random weight neural networks (IR-WNNs) have gained attention in view of its easy implementation and fast learning. However, a significant drawback of IRWNNs is that the relationship between the hidden parameters (node) and the residual error (model performance) is difficult to be interpreted. To address the above issue, this article proposes an interpretable constructive algorithm (ICA) with geometric information constraint. First, based on the geometric relationship between the hidden parameters and the residual error, an interpretable geometric information constraint is proposed to randomly assign the hidden parameters. Meanwhile, a node pool strategy is employed to obtain hidden parameters that is more conducive to convergence from hidden parameters satisfying the proposed constraint. Furthermore, the universal approximation property of the ICA is proved. Finally, a lightweight version of ICA is presented for large-scale data modeling tasks. Experimental results on six benchmark datasets and a numerical simulation dataset demonstrate that the ICA outperforms other constructive algorithms in terms of modeling speed, model accuracy, and model network structure. Besides, two practical industrial application case are used to validate the effectiveness of ICA in practical applications.

 Incremental random weight neural networks, interpretable constructive algorithm, geometric information constraint, universal approximation property, large-scale data modeling.

## I Introduction

Neural networks (NNs), a promising computing paradigm that thoroughly differs from traditional model-based computing, can learn amazingly well patterns from complex data. Therefore, it should not be surprising that NNs are applied in a variety of research fields [1, 2, 3]. The most popular NNs are the deep neural networks (DNNs) and the flat neural networks (FNNs). DNNs realize end-to-end learning by organically combining unsupervised layer-by-layer pre-training with supervised fine-tuning [4]. This learning approach makes DNNs have great potential in expressive power and generalization ability, but it also leads to a time-consuming training process. Recently, due to the universal approximation ability, RWNNs, as a typical representative of FNNs, have been receiving increasing interset [5, 6, 7, 8]. RWNNs are characterized by a two-step training paradigm, i.e., randomly assigning the hidden parameters, and evaluating the output weights by solving a system of linear equations.

Despite the aforementioned advantages, little is known about how network structure for RWNNs fits into modeling tasks. Too large a network structure will result in a poor generalization, while too small a network structure will cause insufficient learning ability. The constructive algorithms always begin with a small network structure (usually a hidden node) and dynamically grow the network structure by adding new hidden nodes incrementally until the requirements are met [9]. This means that the constructive algorithms are likely to offer smaller network structure for modeling tasks [10]. Therefore, constructive version of RWNNs (incremental RWNNs, IRWNNs) have been successfully applied for data modeling tasks [11, 12, 13].

From the perspective of probability theory, the randomly generated hidden parameters are not necessarily suitable for IRWNNs. Therefore, a natural question arises: What hidden parameters are good for IRWNNs? Based on algebraic study of multidimensional nonlinear functions, [14] proved that the relationship between input samples and hidden parameters can be expressed by nonlinear weight equations. [15] showed that there exists a supervisory mechanism between the hidden parameters and the input samples for better network performance. [16] proposed a constructive algorithm with supervisory mechanism to randomly assign hidden parameters in the dynamical interval. [17] proposed a hidden parameters generation approach by analyzing the input samples scope and activation function. Recently, [18] proposed RWNNs with compact incremental inequality constraints, i.e., CIRWN, to improve the quality of hidden parameters. Although these approaches further improve the potential of IRWNNs, very little is known about how hidden parameters accomplish their goals. That is, it is difficult to visualize the influence of each hidden parameter on residual error (network performance). At present, how to interpret the predicted behavior of NNs to improve interpretability is a meaningful and important topic [19, 20]. Thus, further research on interpretable constructive algorithm is important and necessary for RWNNs.
Motivated by the above analysis, this paper proposes an interpretable constructive algorithm (ICA) to help people understand the nature behind the predicted behavior of RWNNs. The main contributions are listed below:

1) The geometric relationship between the hidden parameters and the residual error is employed to build an interpretable geometric information constraint for assigning the randomized hidden parameters in the incremental construction process, and the theoretical analysis is fully discussed.

2) A node pool strategy is developed to further improve the quality of the hidden nodes by searching the hidden parameters that are more conducive to convergence.

3) Using different calculation methods of network output weights, two algorithm implementations, namely ICA and ICA+, are proposed.

The remainder of the article is organized as follows. Section 2 briefly reviews RWNNs and the constructive algorithms. Section 3 proposes an interpretable constructive algorithm and describes it in detail. In section 4, a numerical simulation dataset, six real-world datasets, an ore grinding semi-physical simulation platform, and a gesture recognition system are considered to evaluate the effectiveness and efficiency of the proposed ICA and ICA+. Finally, conclusions are drawn in section 5.

## II Preliminaries

### _Random Weight Neural Networks_

RWNNs can be regarded as a flatted network, where all the hidden parameters (the input weights and biases) are randomly assigned from a fixed interval and fixed during the training process. The output weights are evaluated by solving a system of linear equations. The theory of RWNNs is described as follows.

For a target function \(f:R^{d}\to R^{m}\), the RWNNs with \(L\) hidden nodes can be written as \(:f_{L}=H\beta\), where \(H=\left[g_{1}\left(\omega_{1}^{\mathrm{T}}\cdot x+b_{1}\right),\cdots,g_{L} \left(\omega_{L}^{\mathrm{T}}\cdot x+b_{L}\right)\right]\), \(\mathrm{T}\) denotes matrix transpose, \(x\) is the input sample, \(\omega_{j}\) and \(b_{j}\) are the input weights and biases of the \(j\)-th hidden node, respectively. \(j=1,\cdots,L\), \(g_{j}\) denotes the nonlinear activation function of the \(j\)-th hidden node. The output weights \(\beta\) are evaluated by \(\beta=H^{\dagger}f_{L}\), where \(\beta=\left[\beta_{1},\beta_{2},...,\beta_{L}\right]^{\mathrm{T}}\), \(H^{\dagger}\) denotes the Moore-Penrose generalized inverse of \(H\).

### _Constructive Algorithms_

Constructive algorithms are likely to find the minimal network structure due to their incremental construction nature. Therefore, the constructive algorithms are introduced RWNNs, namely IRWNNs. Specifically, assuming that the IRWNNs with \(L-1\) hidden nodes does not reach the termination condition, then a new hidden node will be generated by the following two steps:

1) The input weights \(\omega_{L}\) and bias \(b_{L}\) are randomly generated from the fixed interval \(\left[-\lambda,\lambda\right]^{d}\) and \(\left[-\lambda,\lambda\right]\). In particular, \(\lambda\) usually takes the value 1. Then, the output vector \(g_{L}\) of the \(L\)-th hidden node, which is determined by maximizing \(\Delta=\frac{\left(e_{L-1,1},g_{L}\right)^{2}}{\left\|g_{L}\right\|^{2}}\), where \(e_{L-1}=f-f_{L-1}=\left[e_{L-1,1},e_{L-1,2},...,e_{L-1,m}\right]\) is the current network residual error. \(f_{L-1}\) is the output of IRWNNs with with \(L-1\) hidden nodes.

2) The output weights vector \(\beta_{L}\) of the \(L\)-th hidden node can be obtained by \(\beta_{L}\)=\(\frac{\left(e_{L-1,1},g_{L}\right)^{2}}{\left\|g_{L}\right\|^{2}}\).

If the new network residual error \(e_{L}=f-f_{L}\) dose not reach the predefined residual error, a new hidden node needs to be added until the predefined residual error or maximum number of hidden nodes is reached.

## III Interpretable Constructive Algorithm

In this section, the interpretable geometric information constraint is constructed based on the geometric relationship between the residual error and the hidden parameters, and the universal approximation property of this constraint is guaranteed by combining the residual error. In addition, a node pool strategy is employed to obtain hidden parameters that are more conducive to convergence. Finally, two different algorithm implementations are proposed, namely ICA and ICA+.

### _Interpretable Geometric Information Constraint_

**Theorem 1**: _Suppose that span(\(\Gamma\)) is dense in \(L^{2}\) and \(\forall g\in\Gamma\), \(0<\left\|g\right\|<v\) for some \(v\in R\). Given \(0<\sigma<1\), \(\sigma=\sigma+rand\left(1-\sigma,1\right)\), \(\tau=\frac{1+\sigma L}{1+L}\) and \(\gamma_{L}\geq\left(1-\tau\right)\). If \(g_{L}\) is randomly generated under interpretable geometric information constraint_

\[\cos^{2}\theta_{L-1}\geq\gamma_{L}\left\langle e_{L-1},e_{L-1}\right\rangle \tag{1}\]

The output weights \(\beta_{L}\) are evaluated by \(\beta_{L}\)=\(\frac{\left(e_{L-1},g_{L}\right)}{\left\|g_{L}\right\|^{2}}\). Then, we have \(\lim_{L\rightarrow+\infty}\left\|e_{L}\right\|=0\).

**Proof:**

Based on the above analysis, we have that

\[\begin{array}{l}\left\|e_{L}\right\|^{2}-\left\|e_{L-1}\right\|^{2}\\ =\left\|e_{L-1}-\beta_{L}g_{L}\right\|^{2}-\left\|e_{L-1}\right\|^{2}\\ =-2\left\langle e_{L-1},\beta_{L}g_{L}\right\rangle+\left\langle\beta_{L}g_{L},\beta_{L}g_{L}\right\rangle\\ =-\frac{\left(e_{L-1},\beta_{L}g_{L}\right)^{2}}{\left\|g_{L}\right\|^{2}}\\ \leq 0\end{array} \tag{2}\]

Then, it has been proved that the residual error \(\left\|e_{L}\right\|\) is monotonically decreasing as \(L\rightarrow\infty\).

It follows from Eq. (1) and Eq. (2) that

\[\begin{array}{l}\left\|e_{L}\right\|^{2}-\tau\left\|e_{L-1}\right\|^{2}\\ =\sum\limits_{q=1}^{m}\left\langle e_{L-1,q}-\beta_{L,q}g_{L},e_{L-1,q}-\beta_{ L,q}g_{L}\right\rangle\\ \quad-\sum\limits_{q=1}^{m}\tau\left\langle e_{L-1,q},e_{L-1,q}\right\rangle\\ =(1-\tau)\sum\limits_{q=1}^{m}\left\|e_{L-1,q}\right\|^{2}-\sum\limits_{q=1}^{m} \frac{\left(e_{L-1,q},g_{L}\right)^{2}}{\left\|g_{L}\right\|^{2}}\\ \leq\gamma_{L}\sum\limits_{q=1}^{m}\left\|e_{L-1,q}\right\|^{2}-\sum\limits_{q=1} ^{m}\frac{\left(e_{L-1,q},g_{L}\right)^{2}}{\left\|g_{L}\right\|^{2}}\\ =\gamma_{L}\|e_{L-1}\|^{2}-\frac{\left(e_{L-1,q}\right)^{2}}{\left\|g_{L}\right\| ^{2}}\end{array} \tag{3}\]
According to \(e_{L}\)=\(e_{L-1}-\beta_{L}g_{L}\) and \(\beta_{L}\)=\(\frac{\left\langle e_{L-1},g_{L}\right\rangle}{\left\|g_{L}\right\|^{2}}\), we have

\[\begin{array}{l}\left\langle e_{L},g_{L}\right\rangle\\ =\left\langle e_{L-1}-\beta_{L}g_{L},g_{L}\right\rangle\\ =\left\langle e_{L-1},g_{L}\right\rangle-\beta_{L}\left\langle g_{L},g_{L} \right\rangle\\ =\left\langle e_{L-1},g_{L}\right\rangle-\frac{\left\langle e_{L-1},g_{L} \right\rangle}{\left\|g_{L}\right\|^{2}}\left\langle g_{L},g_{L}\right\rangle \\ =0\end{array} \tag{4}\]

Then, Eq. (4) means \(e_{L}\)\(\perp\)\(g_{L}\). It can be easily observed that \(e_{L}\), \(e_{L-1}\) and \(\beta_{L}g_{L}\) satisfy the geometric relationship shown in Fig. 1.

In addition, based on \(f_{L-1}=\sum\limits_{j=1}^{L-1}\beta_{j}g_{j}\), we have

\[\begin{array}{l}\sum\limits_{j=1}^{L-1}\beta_{j}\left\langle e_{L-1},g_{j} \right\rangle\\ =\left\langle e_{L-1},\sum\limits_{j=1}^{L-1}\beta_{j}g_{j}\right\rangle\\ =\left\langle e_{L-1},e_{L-1}+f_{L-1}\right\rangle\\ =\left\langle e_{L-1},e_{L-1}\right\rangle+\left\langle e_{L-1},f_{L-1}\right\rangle \\ =\left\langle e_{L-1},e_{L-1}\right\rangle\\ =\left\|e_{L-1}\right\|^{2}\end{array} \tag{5}\]

where \(f_{L-1}\) is orthogonal to \(e_{L-1}\). Thus, we have

\[\begin{array}{l}\exists\beta_{j}\left\langle e_{L-1},g_{j}\right\rangle\geq \frac{\left|e_{L-1}\right|^{2}}{L-1}\\ <=>\beta_{j}\left\|g_{j}\right\|\left\langle\frac{\left\langle e_{L-1},g_{j} \right\rangle}{\left\|g_{j}\right\|}\geq\frac{\left|e_{L-1}\right\|^{2}}{L-1} \\ =>\frac{\left|\left\langle e_{L-1},g_{j}\right\rangle\right|}{\left\|g_{j} \right\|}\geq\frac{\left|e_{L-1}\right\|^{2}}{\left\|e_{L-1}\right\|^{2}} \end{array} \tag{6}\]

where \(\frac{\left\|e_{L-1}\right\|^{2}}{L-1}\)is the average of \(\left\|e_{L-1}\right\|^{2}\).

According to the geometric relationship (Fig. 1) and Eq. (6), the following equation is obtained

\[\frac{\left|\left\langle g_{j},e_{L-1}\right\rangle\right|}{\left\|g_{j} \right\|}=\left(\left\|e_{L-1}\right\|\cos\theta_{L-1}\right) \tag{7}\]

It follows from Eq. (6) and (7) that

\[\cos^{2}\!\theta_{L-1}\geq\varphi\!\left\|e_{L-1}\right\|^{2} \tag{8}\]

where \(0<\varphi=\frac{1}{\left(\left(L-1\right)\beta_{j}\left\|g_{j}\right\|\right)^ {2}}<1\). \(\varphi\) is sufficiently small when \(L-1\) is very large.

Based on the Eq. (3) and Eq. (8), the parameter \(\gamma_{L}\) is directly related to whether the residual error converges. As the modeling process proceeds, the residual error becomes smaller which makes the configuration task on \(w_{L}\) and \(b_{L}\) more challenging. Therefore, the parameter \(\gamma_{L}\) is designed as a dynamic value to ensure that Eq. (9) holds.

\[\varphi\geq\gamma_{L} \tag{9}\]

It follows from Eq. (1), (3), (8), and (9) that

\[\begin{array}{l}\left\|e_{L}\right\|^{2}-\tau\!\left\|e_{L-1}\right\|^{2}\\ \leq\gamma_{L}\!\left\|e_{L-1}\right\|^{2}-\frac{\left\langle e_{L-1},g_{L} \right\rangle^{2}}{\left\|g_{L}\right\|^{2}}\\ \leq\gamma_{L}\!\left\|e_{L-1}\right\|^{2}-\cos^{2}\!\theta_{L-1}\\ \leq 0\end{array} \tag{10}\]

Then, we have that \(\lim_{L\rightarrow+\infty}\left\|e_{L}\right\|=0\).

**Remark 1:** According to Fig. 1, the complex black-box relation between \(e_{L-1}\) and \(g_{L}\) can be visualized using \(\theta_{L}\). Then, Eq. (1) has some interpretability.

### _Node Pool Strategy_

Although the interpretable geometric information constraint can ensures that the constructed network has universal approximation property. However, the hidden parameters (\(g_{L}\)) are generated randomly at a single time, which may not make the network residual decrease quickly. As a result, Eq. (1) is optimized using the node pool strategy as

\[\left(\cos^{2}\!\theta_{L}\right)_{\max}\geq\gamma_{L}\left\langle e_{L-1},e_ {L-1}\right\rangle \tag{11}\]

**Remark 2:** Eq. (11) directly selects the hidden parameter that can minimize the network residual error from many candidates (node pool). However, the traditional output weights calculation method (\(\beta_{L}\)=\(\frac{\left\langle e_{L-1},g_{L}\right\rangle}{\left\|g_{L}\right\|^{2}}\)) may leads to slower convergence. Therefore, two effective methods are designed to evaluate the output weights, namely, 1) global optimization and 2) dynamic stepwise updating.

Fig. 3: Spatial geometry construction process of ICA.

Fig. 2: Network structure of ICA.
### _Algorithm Implementations_

In this section, two different algorithm implementations, termed ICA and ICA+, will be reported. The network structure and spatial geometry construction process of ICA are shown in Fig. 2 and Fig. 3.

#### Iii-C1 Ica

For a target function \(f:R^{d}\to R^{m}\), assume that an ICA with \(L-1\) hidden nodes has been constructed, i.e., \(f_{L-1}=\sum_{j=1}^{L-1}\beta_{j}g_{j}\left(\omega_{j}^{T}\cdot x+b_{j}\right)\). If the generated \(g_{L}\) makes the interpretable geometric information constraint Eq. (11) hold, and the output weights are given by

\[\beta\!\!=\!\arg\min_{\beta}\left\|f-\sum_{j=1}^{L}\beta_{j}g_{j}\right\| \tag{12}\]

Then, we have that \(\lim_{L\rightarrow\infty}\left\|f-f_{L}\right\|\!=\!0\), where \(f_{L}=f_{L-1}+\beta_{L}g_{L}\).

Rearranging Eq.(12), the following matrix form results

\[\beta=H^{\dagger}f_{L} \tag{13}\]

where \(\beta=\left[\beta_{1},\beta_{2},...,\beta_{L}\right]^{\mathrm{T}}\), \(H=\left[g_{1},g_{2},...,g_{L}\right]\), \(H^{\dagger}\) denotes the Moore-Penrose generalized inverse of \(H\).

**Remark 3:** The calculation of the Moore\(-\)Penrose generalized inverse involves the SVD, which will greatly increase computational cost. The problem is even more acute when dealing with large-scale data modeling tasks. To solve this problem, using the iteration theory of Greville [21, 22, 23], the ICA is extended to a lightweight version, called ICA+.

#### Iii-C2 ICA+

For a target function \(f:R^{d}\to R^{m}\), assuming that the ICA with \(L-1\) hidden nodes has been constructed, i.e., \(f_{L-1}=\sum_{j=1}^{L-1}\beta_{j}g_{j}\left(\omega_{j}^{T}\cdot x+b_{j}\right)\). If the generated \(g_{L}\) makes Eq. (11) holds. Let \(H_{L-1}=\left[g_{1},g_{2},...,g_{L-1}\right]\) denote the output weights matrix of the hidden layer with \(L-1\) nodes. \(H_{L}\!\!=\!\!\left[H_{L-1}g_{L}\right]\) represents the output weights matrix of the hidden layer with \(L\) nodes. Based on the iteration theory of Greville, \(H_{L}^{\dagger}\) can be obtained by

\[H_{L}^{\dagger}\!\!=\!\left[\begin{array}{c}H_{L-1}^{\dagger}-d_{L}b_{L}^{ \mathrm{T}}\\ b_{L}^{\mathrm{T}}\end{array}\right] \tag{14}\]

where \(d_{L}=H_{L-1}^{\dagger}g_{L}\), \(c_{L}=h_{L}-H_{L-1}d_{L}\), \(b_{L}^{\mathrm{T}}=\left\{\begin{array}{ccl}\left(c_{L}\right)^{\dagger}&if&c _{L}\neq 0\\ \left(1+d_{L}^{\mathrm{T}}d_{L}\right)^{-1}\!d_{L}^{\mathrm{T}}H_{L-1}^{ \mathrm{T}}&if&c_{L}=0\end{array}\right..\)

Then, the output weights can be derived as

\[\beta\!\!=\!\!\left[\begin{array}{c}\beta^{previous}-d_{L}b_{L}^{\mathrm{ T}}f\\ b_{L}^{\mathrm{T}}f\end{array}\right] \tag{15}\]

where \(\beta^{previous}\) denotes the output weights before a new hidden node is added. The specific implementation steps of ICA and ICA+ are shown in the pseudocode.

```
Initialize\(x=\{x_{1},x_{2},...,x_{N}\}\), \(x_{i}\in R^{d}\) is the input and \(f\!=\!\{f_{1},f_{2},...,f_{N}\}\), \(f_{i}\in R^{m}\) is the output, \(\mathrm{T}_{\mathrm{max}}\) denotes the maximum times of random configuration; \(\mathrm{L}_{\mathrm{max}}\) denotes the maximum number of iteration; \(\ell\) is the expected error tolerance, \(\zeta\!=\!\{\lambda_{\min}:\lambda:\lambda_{\max}\}\) is a set of positive scalars; \(e_{0}=f,0<r<1\), \(\Omega\), \(W\), \(L=1\), \(\Omega\), \(W\), \(L=1\), \(\mathbf{While}\leq L_{\mathrm{max}}\) or \(\left\|e_{0}\right\|>\ell\), do \(\mathbf{HHDHDParameters}\)  For\(\lambda\in\zeta\),do  For\(k=1,2,...,\mathrm{T}_{\mathrm{max}}\),do  Randomly assign the input weights \(\omega_{L}\) and the bias \(b_{L}\) from \(\left[-\lambda,\lambda\right]^{d}\) and \(\left[-\lambda,\lambda\right]\), respectively:  Calculate \(g_{L}\), \(\mathrm{cos}\ell_{L-1}\) based on \(g_{L}=g_{L}\left(\left(\omega_{L}^{T}\cdot x+b_{L}\right)\right)\) and \(\frac{\left\langle e_{L-1}\cdot g_{L}\right\rangle}{\left\|e_{L-1}\right\| \cdot\left\|g_{L}\right\|}\);  Save \(\omega_{L}\) and \(b_{L}\) in \(W\);  End For.  Calculate \(\tau\) and \(\gamma_{L}\);  If \(\left(\mathrm{cos}^{2}\theta_{L-1}\right)\geq\gamma_{L}\)\(\left\langle e_{L-1},e_{L-1}\right\rangle\),do  Find \(w_{L}^{\mathrm{T}}b_{L}^{\mathrm{T}}\) that maximize \(\mathrm{cos}^{2}\theta_{L-1}\) in \(W\);  Set \(H_{L}=\left[g_{1},g_{2},...,g_{L}\right]\);  Else  Renev \(\sigma:=\sigma+rand\left(1-\sigma,1\right)\);  Returns and regenerates the hidden parameters;  End If. Output Weight Evaluation  Calculate \(\beta\) using Eq. (13) (ICA) or Eq. (15) (ICA+);  Obtain \(e_{L}=f-H_{L}\beta\);  Update \(e_{0}=e_{L}\) and \(L=L+1\); End While.
```

**Algorithm** ICA(+)

## IV Experimental Results

In this section, we present the performance of the proposed ICA and ICA+ as well as IRWNNs, and CIRWN on a function approximation dataset, six benchmark datasets, an ore grinding semi-physical simulation platform, and a gesture recognition system. The function approximation dataset was randomly generated by Eq. (16) defined on [0, 1]. The specifications of the seven datasets can be found in TABLE I. In addition, the experimental parameters of all algorithms are summarized in TABLE II. The above experimental parameter settings are the best solutions obtained from multiple experiments.

\[f\left(x\right)=\frac{1}{\left(\left(x-0.3\right)^{2}+0.01\right)}+\frac{1}{ \left(\left(x-0.9\right)^{2}+0.04\right)}-6 \tag{16}\]

where \(x\in\left[0,1\right]\).

All the comparing experiments are implemented on MATLAB 2020a running on a PC with 3.00 GHz Core i7 CPU and 8 GB RAM. Each experiment is repeated 30 times, and the average of the 30 experiments is set as the final reported result. The sigmoid function \(g\left(u\right)=\frac{1}{1+\mathrm{exp}(-u)}\) is employed as the activation function of these four randomized algorithms. In addition, modeling accuracy, root mean squares error (RMSE), and efficiency (the time spent on building the network) were employed to measure the performance of all randomized algorithms.

\[\mathrm{RMSE}=\sqrt{\frac{1}{N}\sum_{i=1}^{N}\left(f_{i}-\tilde{f}_{i}\right)^ {2}} \tag{17}\]

where \(f_{i}\) denotes the real value of the output, \(\tilde{f}_{i}\) is the prediction, \(N\) is the number of the input.
of hidden nodes. In addition, both ICA and ICA+ require fewer hidden nodes to achieve the RMSE convergence. These results show that the proposed two randomized algorithms have obvious advantages in compact structure. Fig. 5 shows the kernel density function (KDF) of the estimated error when the models achieve the expected error tolerance. It can be seen from Fig. 5 that the ICA and the ICA+ perform better than the IRWNNs and the CIRWN due to the KDF of ICA and ICA+ is approximating the real data distribution. This means that the proposed ICA and ICA+ have better prediction ability. Fig. 6 describes the different parameters \(\lambda\) influence on the KDF performance of ICA. It is evident that different \(\lambda\) have favorable or unfavorable effects on the KDF performance of ICA. This shows that \(\lambda\) is an important parameter for the KDF performance of ICA. Therefore, to achieve better KDF performance, \(\lambda\) should not remain fixed.

#### Iv-A2 Benchmark Datasets

In this section, the performance of the ICA, ICA+, IRWNNs, and CIRWN is measured on six benchmark datasets. These benchmark datasets are mainly from KEEL and UCI, and their details can be observed in TABLE I. The experimental parameter information of the four randomized algorithms are given in TABLE II. TABLE III shows the time, training RMSE and testing RMSE results of these four randomized algorithms on six benchmark datasets. As shown in TABLE III, the training RMSE of both ICA and ICA+ is lower than that of CIRWN on most datasets. This means that the interpretable geometric information constraint can help generate better quality hidden parameters. Meanwhile, when compared wth the ICA+, the ICA is weak in the testing and training RMSE performance. This is because ICA+ uses an iterative update method to obtain the output weights, which is very dependent on the quality of the output weights of the first hidden node. In contrast, ICA uses the Moore-Penrose generalized inverse method to obtain the output weights, which enables ICA to obtain the globally optimal output weights after each node is added.

When the same hidden nodes are added, the training time of the proposed ICA and ICA+ is lower than the training time of CIRWN, especially ICA+. Compared with ICA, ICA+ reduced the training time by 14.29%, 50.56%, 67.58%, 43.31%, 67.02%, and 75.96% on Iris, Segment, HAR, Compactiv, Concrete, and Winequality, respectively. Therefore, the LCA+ is superior to other randomized algorithms in terms of lightweight. Fig. 7 depicts the effect of the hidden node pool size \(T_{\max}\) on the RMSE of ICA. It shows that too large or too small \(T_{\max}\) leads to an increase in RMSE. It is worth noting that we do not show the training time. In fact, \(T_{\max}\) is related to the efficiency of the network because it controls the number of hidden parameters obtained from the random interval. In our

Fig. 4: Convergence curve of RMSE for the ICA, ICA+, CIRWN and IRWNNs.

Fig. 5: KDF of the ICA, ICA+, CIRWN and IRWNNs.

Fig. 6: Impact of different \(\lambda\) on KDF.
experiments, this parameter was set with careful trade-offs.

### _Hand Gesture Recognition Case_

Hand Gesture recognition (HGR) is a hot topic in patter recognition due to its wide ranges of applications, such as virtual reality, health monitoring and smart homes [24]. In this section, we evaluate the performance of ICA and ICA+ based on our own developed HGR system. The HGR system framework is shown in Fig. 8, which includes both hardware module and software module. The hardware module consists of a gesture data acquisition and a data transmission shown in Fig. 9. The software module includes a feature extraction and a modeling and recognition shown in Fig. 10. In this section, a gesture dataset with a total sample size of 5136, feature number of 64 and category number of 24 was obtained through the feature extraction of the software module [25]. The dataset has been divided into the training dataset and the testing dataset.

#### Iv-B1 Parameter Configuration

For IRWNNs, the random set of hidden parameters is fixed interval [-150,150]. The random parameters of the other three randomized algorithms is selected from a variable interval \(\zeta\)=\(\{150:10:200\}\). For ICA, ICA+, and CIRWN, the maximum number of iteration is set to \(L_{\max}\) = 500, and the maximum times of random configuration is set to \(T_{\max}\) = 20.

#### Iv-B2 Comparison and Discussion

The average RMSE of four algorithms based on thirty times of experiments on the HGR testing dataset is displayed in Fig. 11. It can be found that the ICA and ICA+ have good stability performance in terms of RMSE. As can be seen from Fig. 11, the difference between the maximum RMSE and the minimum RMSE for IRWNNs and CIRWN is 0.15 and 0.02, respectively. TABLE IV shows the experimental results of the IRWNNs, ICA, ICA+, and CIRWN on the HGR system. It can be seen from TABLE IV that compared with IRWNNs and CIRWN, the ICA and ICA+ have a great advantage in terms of training time and classification accuracy. Based on the comparisons and analyzes of these results, we can conclude that the proposed ICA and ICA+ are more effective than IRWNNs and CIRWN for HGR tasks.

Fig. 8: Framework diagram of gesture recognition system.

Fig. 10: Software module.

Fig. 7: Impact of node pool on RMSE.

Fig. 9: Hardware module.
### _Ore grinding Case_
## V Conclusion

In this paper, an interpretable constructive algorithm (ICA) is proposed to visualize the contribution of each hidden parameter on residual error to improve the interpretability of RWNNs predicted behavior. In ICA, the hidden parameters are randomly assigned by the interpretable geometric information constraint with node pool strategy. Further, ICA is extended to ICA+ in order to reduce the computational cost. In particular, the difference between ICA+ and ICA is that ICA+ uses a more lightweight and efficient iterative update method to evaluate the output weights, while ICA uses a globally optimal approach to evaluate the output weights. Experimental results on seven benchmark datasets, a hand gesture recognition system and an ore grinding semi-physical simulation platform show that ICA and ICA+ can effectively reduce computational consumption and have better network performance than other construction algorithms.
