Connected Hidden Neurons (CHNNet): An Artificial Neural Network for
  Rapid Convergence
Despite artificial neural networks being inspired by the functionalities of
biological neural networks, unlike biological neural networks, conventional
artificial neural networks are often structured hierarchically, which can
impede the flow of information between neurons as the neurons in the same layer
have no connections between them. Hence, we propose a more robust model of
artificial neural networks where the hidden neurons, residing in the same
hidden layer, are interconnected that leads to rapid convergence. With the
experimental study of our proposed model in deep networks, we demonstrate that
the model results in a noticeable increase in convergence rate compared to the
conventional feed-forward neural network.
Rafiad Sadat Shahir, Zayed Humayun, Mashrufa Akter Tamim, Shouri Saha, Md. Golam Rabiul Alam
# Connected Hidden Neurons (CHNNet): An Artificial Neural Network for Rapid Convergence

###### Abstract

The core purpose of developing artificial neural networks was to mimic the functionalities of biological neural networks. However, unlike biological neural networks, traditional artificial neural networks are often structured hierarchically, which can impede the flow of information between neurons as the neurons in the same layer have no connections between them. Hence, we propose a more robust model of artificial neural networks where the hidden neurons, residing in the same hidden layer, are interconnected, enabling the neurons to learn complex patterns and speeding up the convergence rate. With the experimental study of our proposed model as fully connected layers in shallow and deep networks, we demonstrate that the model results in a significant increase in convergence rate.

## 1 Introduction

The brain, which is the main inspiration behind developing artificial neural networks, processes large amounts of data passed by senses from different parts of the body. The neurons in the brain form complex and dense connections among themselves, which is important for efficient and flexible information processing[25]. A brain can have approximately 100 billion neurons and 100 trillion neuronal connections, which implies that each neuron can have connections with 1 thousand other neurons[2]. Artificial neural networks (ANNs), on the contrary, often follow hierarchical structures with simple neural connections, which can impede the flow of information between neurons as the neurons in the same layer have no connections between them. In some scenarios, to improve the generalization power of new and unseen data, it is important to have more connections among the neurons, as a network with more connections can learn more robust and meaningful features[29]. Moreover, having more connections among the neurons can potentially speed up the convergence rate, as it helps to learn complex patterns and relations in the data[5].
We hypothesize that designing a neural network model with an increased number of neural connections would result in a performance gain. In traditional ANNs, specifically in multilayer perceptrons (MLP), to increase the number of connections while keeping the number of layers fixed, the number of neurons per hidden layer has to be increased[5]. However, increasing the number of neurons can lead to overfitting and a slow convergence problem in the model[6]. To prevent overfitting and improve the generalization performance of MLPs, regularization techniques such as dropout[26] and weight decay[24] are used. On the other hand, to solve the problem of slow convergence, activation functions such as the rectified linear unit (ReLU)[4], techniques such as batch normalization[14] and adaptive gradient methods such as the Adam optimizer[16] are used. However, using regularization[29], activation functions[3], batch normalization[13] and adaptive gradient methods[27] have some undesirable limitations as well. Hence, we propose to connect the hidden neurons of the networks in order to increase the number of neural connections in the network. However, connecting all the hidden neurons in a network will be very compute-intensive, and thus we design an ANN model where the hidden neurons, residing in the same hidden layer only, are interconnected.

The primary contributions of the paper are summarized as follows:

* We introduced a neural network model, namely CHNNet (Connected Hidden Neurons), in which we created connections among the hidden neurons residing in the same hidden layer, enabling robust information sharing among the neurons.
* We formulated mathematical equations to calculate the activations of the hidden layers in forward propagation and revised the backpropagation algorithm to calculate the gradients based on the formulated forward propagation equations, preserving the parallel computability of the model.
* We trained the proposed model on benchmark datasets and demonstrated its performance using a variable number of hidden layers, hidden neurons, and epochs against the traditional MLP model. The model depicted a significant increase in convergence rate without showing overfitting behavior.
* As our model generates a larger number of trainable parameters, we tested the model against that of a traditional MLP model. The MLP model's trainable parameters were increased to be around the same as CHNNet. Our results showed that CHNNet outperformed the MLP model, particularly in deep networks.

## 2 Literature review

In the infancy of neural networks, Minsky and Papert [21] specified significant drawbacks of perceptrons and suggested the raw idea of MLP. The MLP architecture they proposed is hierarchical in structure and has no connections among hidden neurons in the same layer. Further, a few MLP architectures were analyzed in the literature by Rumelhart et al. [23], none of which featured connections among the hidden neurons.

Thus far, a number of ANNs have been introduced using different approaches to establish connections among neurons. A Recurrent Neural Network (RNN) has self-connections among hidden neurons through time; that is, the self-connections work as information carriers from one time step to another, which is best suited for processing sequential data [23]. The Kohonen Neural Network, proposed by Kohonen [17], has neurons arranged in a two-dimensional lattice structure and strongly connected to neighboring neurons, which enables the network to perform better on data clustering problems. The Hopfield Neural Network, a single-layered neural network introduced by Hopfield [11], has neurons symmetrically connected to all other neurons through bidirectional connections and is primarily used for associative memory problems. Similar to the Hopfield Network, the Boltzmann Machine has its neurons connected symmetrically with all other neurons, with the exception that the neurons are divided into visible units and hidden units [10]. Neural networks like the Echo State Network (ESN) [15] and Liquid State Machine (LSM) [20] have featured a pool of neurons, namely a reservoir, which consists of numerous randomly connected neurons, providing them with non-linear modeling ability. However, as the reservoir is randomized, it requires numerous trials and sometimes even luck [22]. It should be emphasized that the referred ANNs have specific use cases that are irrelevant to the concentration of our work.
In the contemporary period, designing new paths for information flow in neural networks has attained noticeable success. Convolutional Neural Network (CNN) architectures like DenseNet [12], ResNet [9] and UNet++[30], which use skip connections to directly pass information from a layer to a deeper layer, have reached state-of-the-art (SOTA) performance. Moreover, Liu et al. [19] introduced the Group Neural Network, which, to overcome the blockade at information passing, features a group of neurons that can connect freely with each other. However, due to its irregular architecture, the training of the network cannot be accelerated through parallel computing. It is significant to mention that our approach to connecting the hidden neurons is distinct from that of the referred ANNs.

## 3 Methodology

The proposed architecture features additional self-connections and interconnections among the hidden neurons, as shown in figure 1.

We formulated mathematical equations for forward propagation and revised the backpropagation algorithm for hidden layers only, as no new connections are introduced in input and output layers.

### Forward propagation

Rumelhart et al. [23] note that the process of calculating the activations of each layer in the forward direction is straightforward and can be done quickly using matrix operations. Therefore, the forward propagation can be computed for an entire batch of inputs using matrix operations, which is much faster than computing it for each input individually. Mathematically, forward propagation can be expressed as follows:

Let \(Y\) be the output, and \(f\) be the activation function. For the \(l\)th hidden layer, the input is \(a^{[l-1]}\) and the output \(a^{[l]}\) is computed as:

\[z^{[l]}=w^{[l]}a^{[l-1]}+b^{[l]}\]

\[a^{[l]}=f(z^{[l]})\]

where \(w^{[l]}\) is the weight matrix connecting \((l-1)\)th layer to \(l\)th layer, \(b^{[l]}\) is the bias vector \(l\)th layer and \(z^{[l]}\) and \(a^{[l]}\) are the pre-activation and post-activation outputs of \(l\)th layer respectively. The output of the network is computed as:

\[Y=f(z^{[L]})\]

where \(z^{[L]}\) is the pre-activation output of the output layer and \(L\) is the number of layers in the network.

Unlike traditional MLP architectures, in CHNNet, information from one hidden neuron is being incorporated into the other hidden neurons residing in the same hidden layer. Therefore, for the forward propagation, we have two sets of weight matrices, one set is the weight matrix connecting \((l-1)\)th layer to \(l\)th layer and the other set is the weight matrix connecting hidden neurons of \(l\)th

Figure 1: Proposed architecture of CHNNet with (a) one hidden layer and (b) two hidden layers.
layer to other hidden neurons of the layer. Then for layer \(l\), the input is \(a^{[l-1]}\) and the pre-activation output \(z^{[l]}\) is computed as:

\[z^{[l]}=w_{1}^{[l]}\cdot a^{[l-1]}+w_{2}^{[l]}\cdot h^{[l]}+b^{[l]} \tag{1}\]

where \(w_{1}^{[l]}\) is the weight matrix connecting \((l-1)\)th layer to \(l\)th layer, \(w_{2}^{[l]}\) is the weight matrix connecting hidden neurons of \(l\)th layer to other hidden neurons of the layer, \(b^{[l]}\) is the bias vector \(l\)th layer, \(h^{[l]}\) is the post-activation output of the hidden neurons and \(z^{[l]}\) and \(a^{[l]}\) are the pre-activation and post-activation outputs of \(l\)th layer respectively.

The post-activation output of the hidden neurons \(h^{[l]}\) in equation 1 is the variable that is introduced in CHNNet. For \(l\)th layer, the input is \(a^{[l-1]}\) and output \(h^{[l]}\) is computed as:

\[t^{[l]}=w_{1}^{[l]}\cdot a^{[l-1]}+b^{[l]} \tag{2}\]

\[h^{[l]}=f(t^{[l]}) \tag{3}\]

where \(t^{[l]}\) is the pre-activation output of \(l\)th layer. Finally, the post-activation output \(a^{[l]}\) of \(l\)th layer is computed as:

\[a^{[l]}=f(z^{[l]})\]

### Backpropagation

As described by Rumelhart et al. [23], the backpropagation can be mathematically expressed as follows:

Let \(Y\) be the output, \(f\) be the activation function and \(E\) be the loss function. The upstream gradient for the output layer \(\delta_{u}^{(L)}\), where \(L\) is the number of layers in the network, can be computed as:

\[\delta_{u}^{(L)}=\frac{\partial E}{\partial Y}\]

Now for \(l\)th layer, the upstream gradient \(\delta_{u}^{[l]}\) can be computed as:

\[\delta_{u}^{[l]}=w^{[l+1]^{\mathsf{T}}}\cdot\delta^{[l+1]}\]

where \(w^{[l+1]}\) is the weight matrix connecting \((l)\)th layer to \((l+1)\)th layer and \(\delta^{[l+1]}=\frac{\partial E}{\partial z^{[l+1]}}\), where \(z^{[l]}\) be the pre-activation output of \(l\)th layer. Then for \(l\)th layer \(\delta^{[l]}\) can be computed as:

\[\delta^{[l]}=\delta_{u}^{[l]}\odot f^{\prime}(z^{[l]})\]

In the end, the gradients can be calculated as:

\[\frac{\partial E}{\partial w^{[l]}}=\delta^{[l]}\cdot a^{[l-1]^{\mathsf{T}}}\]

\[\frac{\partial E}{\partial b^{[l]}}=\delta^{[l]}\]

Finally the weight matrix \(w^{[l]}\) and bias vector \(b^{[l]}\) can be updated using the following equations:

\[w^{[l]}\to w^{[l]}-\eta\cdot\frac{\partial E}{\partial w^{[l]}} \tag{4}\]

\[b^{[l]}\to b^{[l]}-\eta\cdot\frac{\partial E}{\partial b^{[l]}} \tag{5}\]

In contrast to traditional MLP architectures, CHNNet has two sets of weight matrices. The weight matrix \(w_{1}^{[l]}\) can be updated using equation 4. For weight matrix \(w_{2}^{[l]}\), the gradient can be computed as:

\[\frac{\partial E}{\partial w_{2}^{[l]}}=\delta^{[l]}\cdot{h^{[l]}}^{\mathsf{T}} \tag{6}\]

Finally, weight matrix \(w_{2}^{[l]}\) can be updated using as follows:

\[w_{2}^{[l]}\to w_{2}^{[l]}-\eta\cdot\frac{\partial E}{\partial w_{2}^{[l]}} \tag{7}\]

The process of backpropagation for a hidden layer is summarized in Algorithm 1.
```
Computing Delta: \[\delta^{[l]}=\delta^{[l]}_{u}\odot f^{\prime}(z^{[l]})\] Computing Gradients: \[\frac{\partial E}{\partial w^{[l]}}=\delta^{[l]}\cdot a^{[l-1]^{\mathsf{T}}}\] \[\frac{\partial E}{\partial w^{[l]}_{2}}=\delta^{[l]}\cdot h^{[l]^{ \mathsf{T}}}\] \[\frac{\partial E}{\partial b^{[l]}}=\delta^{[l]}\] Updating Weights: \[w^{[l]}_{1}\to w^{[l]}_{1}-\eta\cdot\frac{\partial E}{\partial w^{[l]}_{1}}\] \[w^{[l]}_{2}\to w^{[l]}_{2}-\eta\cdot\frac{\partial E}{\partial w^{[l]}_{2}}\] \[b^{[l]}\to b^{[l]}-\eta\cdot\frac{\partial E}{\partial b^{[l]}}\] Computing Downstream Gradient: \[\delta^{[l-1]}_{u}=w^{[l]^{\mathsf{T}}}_{1}\cdot\delta^{[l]}\]
```

**Algorithm 1** Backpropagation for a single hidden layer in CHNNet

## 4 Performance evaluation

To evaluate the performance of our proposed model, we used the software library TensorFlow 2 by Google. Using the TensorFlow library, we constructed a layer, namely the CHN Layer, implementing the forward and backpropagation described in Section 3. Using the CHN layer, along with other layers provided by the TensorFlow library, we performed all our experiments. Our goal was not to get SOTA performance on the benchmark datasets. Rather, it was to achieve a better convergence rate without overfitting behavior. We compared the performance of the CHN layer, as fully connected hidden layers in shallow networks and deep networks, with that of the Dense layer provided by the TensorFlow library, which implements traditional MLP forward propagation and backpropagation mechanisms.

### Datasets

We evaluated the performance of the CHN layer on six benchmark datasets of different sizes and diverse features. All the chosen datasets are well-structured, with clear patterns and few outliers.

For our experiments in shallow networks, we used three benchmark datasets, namely Abalone, Iris and the MNIST dataset. The Abalone dataset, used for linear regression experiments, contains 4177 instances and 8 attributes[1]. The Iris dataset contains 150 instances of 3 classes of iris plants, each having 4 numerical attributes[1]. We split both the Abalone and Iris datasets in a ratio of 80:20 randomly for training and testing sets, respectively. The MNIST dataset, consisting of 28x28 images of handwritten digits divided into 10 classes, holds 60,000 training samples and 10,000 testing samples[18]. Both the Iris and MNIST datasets are used for classification experiments.

We used another three benchmark datasets to conduct our experiments in deep networks, namely the Boston Housing, Breast Cancer and Fashion MNIST datasets. The Boston Housing dataset, used for linear regression experiments, contains 506 instances and 13 attributes[7]. The Breast Cancer (Diagnostics) dataset contains 569 instances and 30 numerical attributes of 2 classes of breast cancer [31]. We split both the Boston Housing and Breast Cancer datasets in a ratio of 80:20 randomly as well. The Fashion MNIST dataset has the exact same features as the MNIST dataset mentioned earlier, with the only exception that instead of digits, the dataset has 10 classes of fashion accessories[28]. Both the Breast Cancer and Fashion MNIST datasets are used for classification experiments.
### Hyperparameters

We chose the number of hidden neurons in the hidden layers to be a power of 2 due to hardware reasoning. The loss functions for the experiments were chosen depending on the type of dataset and the desired output format. We used mean square error in linear regression problems [8], categorical cross entropy in multi-class classification problems [5] and binary cross entropy in binary classification problems [5]. The optimizers are selected based on the improvement they could bring to the performance of the models.

### CHN layer in shallow networks

#### 4.3.1 Training parameters

We trained the models on the Abalone dataset using a network with 8-1 neurons, mean square error as a loss function and batches of size 256. For the Iris dataset, we used a network as 128-3, categorical cross entropy as a loss function and one batch of size 120. As optimizer, for both the Abalone and Iris data sets, we used the Adam optimizer with a learning rate of 0.001. While training on the MNIST dataset, we used a 128-10 network, an RMSprop optimizer with a learning rate of 0.001, sparse categorical cross entropy as a loss function and batches of size 512.

#### 4.3.2 Test results

As shown in figure 2, the CHN layer had a better convergence rate, specifically on the Iris and MNIST datasets, compared to the Dense layer. Moreover, when trained using a smaller number of epochs on the Iris and MNIST datasets, CHNNet was able to achieve more accuracy than the Dense layer, as demonstrated in table 1. On the abalone dataset, the CHN layer had comparable performance to the Dense layer, as there was no significant performance difference between the models. However, the CHN layer generated more trainable parameters than the Dense layer in all experiments, as new connections among the hidden neurons were introduced in the designed layer.

### CHN layer in deep networks

#### 4.4.1 Training parameters

We used a network with 64-64-1 neurons, an RMSprop optimizer with a learning rate of 0.001 and the mean square error as the loss function for training the models on the Boston Housing dataset. For

Figure 2: Loss graph of CHN layer and Dense layer on (a) Abalone, (b)Iris and (c)MNIST dataset and accuracy graph of the layers on (d)Iris and (e)MNIST dataset.
the Breast Cancer dataset, we used a network of 64-64-3, an Adam optimizer with a learning rate of 0.001 and binary cross entropy as a loss function. For experiments on both the Boston Housing and Breast Cancer datasets, we used batches of size 128. While training on the Fashion MNIST dataset, we used a 256-256-256-10 network, an SGD optimizer with a learning rate of 0.001, sparse categorical cross entropy as a loss function and batches of size 32.

#### 4.4.2 Test results

Compared to shallow networks, the CHN layer showed significantly better performance in deep networks, with faster convergence as depicted in figure 3 and better loss and accuracy measurements than the Dense layer when tested with a smaller number of epochs, as illustrated in table 1. Specifically, on the Fashion MNIST dataset, the CHN layer noticeably outperformed the Dense layer with more accuracy and less loss for any number of epochs. However, in deep networks as well, the CHN layer generated more trainable parameters than the Dense layer.

### Equating CHN layer and Dense layer

In the experiments conducted previously, the CHN layer generated more trainable parameters than the Dense layer. Thereby, we conducted some more experiments with an increased number of hidden

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & & \multicolumn{3}{c}{Dense Layer} & \multicolumn{3}{c}{CHN Layer} \\ \hline Datasets & epoch & Trainable & Average & Average & Trainable & Average & Average \\  & & Params & Loss & Accuracy & Params & Loss & Accuracy \\ \hline Abalone & 50 & 97 & 0.007 & - & 161 & 0.007 & - \\  & 20 & 97 & 0.012 & - & 161 & 0.009 & - \\ \hline Iris & 500 & 1,027 & 0.16 & 93.33 & 17,411 & 0.21 & 93.33 \\  & 50 & 1,027 & 0.54 & 84.34 & 17,411 & 0.27 & **92.15** \\ \hline MNIST & 30 & 101,770 & 0.08 & 97.68 & 118,154 & 0.09 & 97.67 \\  & 5 & 101,770 & 0.12 & 96.21 & 118,154 & 0.093 & **97.15** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance measurement of CHN layer in shallow network.

Figure 3: Loss graph of CHN layer and Dense layer on (a) Boston Housing, (b)Breast Cancer and (c)Fashion MNIST dataset and accuracy graph of the layers on (d)Breast Cancer and (e)Fashion MNIST dataset.
neurons in the dense layer to raise the number of trainable parameters in the Dense layer around that of the CHN layer.

#### 4.5.1 Training parameters

All the training parameters were exactly the same as in previous experiments, except that we increased the number of hidden neurons in the Dense layer. Hence, we had Dense layers with an architecture of 16-1 for the abalone dataset, 2048-3 for the iris dataset, 150-10 for the MNIST dataset, 110-110-1 for the Boston Housing dataset, 105-105-1 for the Breast Cancer dataset and 360-360-360-10 for the fashion MNIST dataset.

#### 4.5.2 Test results

Even after increasing the number of hidden neurons in Dense layers, the CHN layer showed a significantly faster convergence rate without acquiring overfitting behavior on the Boston Housing, Breast Cancer and Fashion MNIST datasets. As shown in table 3, the CHN layer outperformed the Dense layer in those datasets as well. It should be emphasized that the proposed layer exhibits better performance in deep networks compared to shallow networks. In shallow networks as well, the CHN layer exhibits comparable performance to the Dense layer.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & & \multicolumn{3}{c}{Dense Layer} & \multicolumn{3}{c}{CHN Layer} \\ \hline Datasets & epoch & Trainable & Average & Average & Trainable & Average & Average \\  & & Params & Loss & Accuracy & Params & Loss & Accuracy \\ \hline Boston & 100 & 5,121 & 26.77 & 31.84 & 13,313 & 25.21 & - \\  & 50 & 5,121 & 50.71 & - & 13,313 & **25.45** & - \\ \hline Cancer & 100 & 6,209 & 0.1 & 96.76 & 14,401 & 0.05 & 97.35 \\  & 30 & 6,209 & 0.13 & 95.65 & 14,401 & 0.06 & **97.1** \\ \hline F\_MNIST & 10 & 335,114 & 0.365 & 86.47 & 531,722 & 0.332 & **87.81** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance measurement of CHN layer in deep network.

Figure 4: Loss graph of CHN layer and Dense layer on (a) Abalone, (b)Iris, (c)MNIST, (d)Boston Housing, (e)Breast Cancer and (f)Fashion MNIST dataset.
### Discussion

From the experiments conducted, it is evident that CHNNet has the potential to outperform the Dense layer. Specifically, in deep networks, CHNNet depicts promising outcomes. Though CHNNet generates a larger number of trainable parameters than traditional MLP models, with nearly equal numbers of trainable parameters, the proposed model exhibits comparable performance in shallow networks and outperforms the traditional MLP model in deep networks.

## 5 Future Work

CHNNet, thus far, has been tested on a relatively small set of benchmark datasets, which leaves the scope for more comprehensive experiments with the model. As the model generates numerous parameters, some algorithms can also be proposed to reduce the number of connections systematically. Moreover, the model has not been tested as a fully connected layer in CNN architectures, which also has the potential to generate compelling outcomes. It would be interesting to see if more efficient CNN architectures could be developed utilizing the features of CHNNet. Further, there is the opportunity for research on implementing RNN models based on architecture of CHNNet.

## 6 Conclusion

We designed a rapidly converging ANN, namely CHNNet, which is different from the existing multi-layered networks in that it creates connections among the hidden neurons of the same layer. In addition, we described the forward propagation and backpropagation mechanisms of the proposed model. CHNNet depicted a significant increase in convergence rate, resulting in fewer epochs for training compared to traditional MLP-based models, especially in deep networks. On the contrary, the model generated numerous trainable parameters, which increased the training time per epoch compared to the traditional models. However, even with a nearly equal number of trainable parameters in both CHNNet and traditional models, CHNNet depicted a comparable performance in shallow networks and outperformed the traditional models in deep networks without adapting overfitting behavior.
