Mitigating Communication Costs in Neural Networks: The Role of Dendritic
  Nonlinearity
Our comprehension of biological neuronal networks has profoundly influenced
the evolution of artificial neural networks (ANNs). However, the neurons
employed in ANNs exhibit remarkable deviations from their biological analogs,
mainly due to the absence of complex dendritic trees encompassing local
nonlinearity. Despite such disparities, previous investigations have
demonstrated that point neurons can functionally substitute dendritic neurons
in executing computational tasks. In this study, we scrutinized the importance
of nonlinear dendrites within neural networks. By employing machine-learning
methodologies, we assessed the impact of dendritic structure nonlinearity on
neural network performance. Our findings reveal that integrating dendritic
structures can substantially enhance model capacity and performance while
keeping signal communication costs effectively restrained. This investigation
offers pivotal insights that hold considerable implications for the development
of future neural network accelerators.
Xundong Wu, Pengfei Zhao, Zilin Yu, Lei Ma, Ka-Wa Yip, Huajin Tang, Gang Pan, Tiejun Huang
# Mitigating Communication Costs in Neural Networks: The Role of Dendritic Nonlinearity

###### Abstract

Our comprehension of biological neuronal networks has profoundly influenced the evolution of artificial neural networks (ANNs). However, the neurons employed in ANNs exhibit remarkable deviations from their biological analogs, mainly due to the absence of complex dendritic trees encompassing local nonlinearity. Despite such disparities, previous investigations have demonstrated that point neurons can functionally substitute dendritic neurons in executing computational tasks. In this study, we scrutinized the importance of nonlinear dendrites within neural networks. By employing machine-learning methodologies, we assessed the impact of dendritic structure nonlinearity on neural network performance. Our findings reveal that integrating dendritic structures can substantially enhance model capacity and performance while keeping signal communication costs effectively restrained. This investigation offers pivotal insights that hold considerable implications for the development of future neural network accelerators.
## 1 Introduction

In the past decade, we have observed a remarkable increase in artificial neural network (ANN) utilization across different domains. To some extent, this gives us an impression that AI is closing in on human-level intelligence [1, 2]. Looking back to the beginning of neural networks, we will find out that those ANNs were structured to mimic the neuronal networks in our brains. However, it is crucial to acknowledge that neurons in contemporary ANNs exhibit considerable differences from their biological counterparts. The following equation can represent a typical ANN neuron:

\[h=\sigma(\sum_{i=1}^{n}w_{i}x_{i}+b)\,. \tag{1}\]

Here, \(\sigma\) denotes the nonlinear output function, \(w_{i}\) and \(x_{i}\) correspond to the weights and inputs, and \(b\) is the bias term. These neurons, commonly called point neurons, are characterized by their simple weighted summation properties, which contrast with the intricate dendritic structures observed in biological neurons, as illustrated in Fig. 1.

The dendritic structure is indispensable in neuronal network computation because it offers a better surface-area-to-volume ratio [7, 8]. Unlike cells with a more compact shape, this attribute enables neurons to gather synaptic inputs through their branched and elongated form effectively. In contemporary ANN models processed on general-purpose computing hardware, such as CPUs and GPUs, a physical dendrite structure is no longer required to enhance information collection efficiency.

Suppose a dendritic structure is not needed for collecting synaptic inputs, and point neuron-based ANN models have been quite successful in the last decade. Is the dendritic structure just a fancy decoration no longer necessary for our modern ANNs? Evidence suggests that localized nonlinear signal processing happens inside the dendritic tree. Therefore we cannot dismiss whether dendrites should play a significant role for ANNs yet.

In recent decades, experimental and computational neuroscience research has curated strong evidence that dendrites actively funnel synaptic inputs toward cell bodies instead of passively. Dendrites are active because their membrane is embedded with many voltage-gated ion channels, for example, voltage-gated sodium, calcium, and NMDA channels [9, 10, 11, 12]. Those channels lead to the nonlinearity of the dendritic input-output function. Earlier studies have assigned many different roles to the active dendrites, including counter-balance spatial attenuation at the distal end of dendrites, improving model expressivity, enabling efficient learning, and enabling dendrites to detect temporal sequences [7, 13, 14, 15, 16, 17, 18].

We can assign many more roles to the active dendrites with diverse signal-processing functions. However, it is well established that any nonlinear function performed by active dendrites can be replicated by a series of point neurons, as
per the Universal Approximating Theory [19]. Given this, the question remains: Why are dendrites necessary in the nervous system, Are dendrites relevant for ANNs?

We endeavor to address this crucial question via a machine-learning perspective. Through our analysis, we identify the central role of active dendrites to efficiently enhance model capacity without incurring excessive communication overhead. Notably, recent research has underscored communication as the predominant factor in energy consumption for both ANNs and biological neural networks during computation, as highlighted in the works of Dally et al. [20] and Levy et al. [21]. Our study illuminates the pivotal role of dendrites within neuronal networks, offering insights with significant implications for practical applications in a real-world context.

Figure 1: (**A, B, C**) Illustration of three representative neurons showcasing distinct dendritic structures from left to right: A chicken bipolar neuron [3], a human hippocampal pyramidal neuron [4], and a ferret neocortical pyramidal neuron [5]. All neuron models were derived from the Neuromorpho.org database [6]. (**D**) Portrays a layer of a neural network made up of point neurons, as characterized by Equation 1. (**E**) Illustrates a comparable network layer, but composed of dendritic neurons as detailed by Equation 3. An exemplary dendritic neuron is highlighted within the red dotted line for clearer understanding.
## 2 Results

In this study, we aim to identify the primary role played by the active dendritic structure of neurons. To this end, we reduce dendritic structure to a dual-layered neural network [11, 22], as exemplified in Fig. 1E. The mathematical representation of the simplified dendritic neuron employed in our study is provided by the equations (Eqs. 2, 3) shown below. The first line of the expression defines the dendritic computation, where \(W_{i}\) represents the weight vector for a specific dendrite and \(X\) denotes the input activation vector. The second line of the expression illustrates that the outputs of every \(K\) dendrites are aggregated to yield the output of a neuron.

\[d_{i} =\sigma(W_{i}X+b_{i})\,, \tag{2}\] \[h =\theta(\sum_{i=1}^{K}d_{i})\,. \tag{3}\]

In the proposed architectural framework, incoming data is initially integrated at each dendrite before undergoing a transformation via a nonlinear function, denoted as \(\sigma\). Subsequently, the outputs of these nonlinear functions are aggregated and, if necessary, further processed by an optional nonlinear function represented by \(\theta\) (not used in this study). This refined output is then transmitted to the downstream recipients.

It is imperative to note that while each dendritic unit possesses a similar information-processing capacity to a point neuron, a distinct divergence exists in how their respective outputs are conveyed to downstream neurons. Contrary to a point neuron's output, which is independently channeled to the downstream neurons, dendritic outputs necessitate sharing a common channel with fellow dendrites of the same neuron for information dispatch.

### Dimension expansion with active dendrites

Before exploring the experimental aspects in-depth, we want to first develop an intuitive comprehension of dendrites and their significance in biological brains, particularly from a machine-learning standpoint.

Dimension expansion is an essential technique within the machine learning domain, which facilitates the mapping of original input data into an alternate basis within a higher-dimensional space, thereby enhancing pattern separation capabilities [23, 24]. It is postulated that this methodology bears a striking resemblance to strategies employed within biological brains, such as in the vertebrate cerebellum [24]. In this context, a relatively limited number of mossy fibers project onto a substantially larger number of granule cells, as demonstrated in studies by [25, 26]. Analogously, a similar expansion of inputs can be observed in various sensory pathways, such as in the case of cats, where the
input signals from the lateral geniculate nucleus to the V1 cortex undergo a 25-fold expansion [27, 28].

On the other hand, through empirical exploration, researchers and practitioners of deep neural networks have discovered that scaling up networks is the core receipt to achieve good model performance [29]. The mechanism behind this scaling behavior remains elusive, with dimension expansion potentially playing a role. Both expanding feature dimensionality and model capacity are associated with the high costs in biological and artificial neural networks. In the case of the mossy fiber projection to granule cells, granule cells account for 99% of all neurons in the cerebellum [26, 30].

The scaling behavior observed in artificial neural networks has led to the adoption of large models such as GPT-3 [31], ChatGPT [32] and MT-NLG [33]. With the advent of these greatly expanded models, both biological and artificial neural networks face increased costs. In biological brains, more synapses and neurons are required to carry out tasks. In artificial neural networks, larger memory space is required to store weights and intermediate activation values. More computing hardware is needed to handle the expanded computing needs, resulting in a high energy cost.

### Reducing communication cost

The energy required for computing in neural networks is considerable, yet it is not the primary cost involved. The dominant cost in the computing process of neural networks stems from communication rather than computing itself. In biological brains, only a small fraction of energy is spent on the computing part. As highlighted by Levy et al. [21], the communication process consumes 35 times more energy than the computing part of the brain.

To put things in perspective, communication costs in artificial neural networks can be orders of magnitude higher than computing costs. For instance, the energy cost of adding two 32-bit numbers may only be 20 femtojoules (fJ), but fetching those two numbers from memory can consume 1.3 nanojoules (nJ). This means that the communication process in this example consumes 64,000 times more energy than the computing process [20].

Artificial neural networks must grapple with controlling communication costs like their biological counterparts. By examining biological neuronal networks, we can glean strategies to minimize these costs in artificial systems. Our study demonstrates that incorporating active dendrites can play a significant role in addressing this issue.

### Evaluating efficiency of dendritic structure

To bolster a neural network's capacity for encoding information and enhancing expressivity, a prevalent technique involves widening the network, specifically by adding more neurons to hidden layers. This approach has been consistently demonstrated to increase a model's capacity and improve its generalization
Figure 2: Comparison of Resnet-18-style models composed of point and dendritic neurons trained on the ImageNet dataset. Each experiment was performed 5 times, with standard deviations displayed. (A) Training loss values, (B) Training accuracy, and (C) Test accuracy for models with varying numbers of dendrites per neuron at four distinct levels of network width. \(x\)-axis indicates the number of dendrites per neuron; models with one dendrite per neuron are point neuron-based models.
performance. As previously noted, communication energy consumption constitutes a significant cost in neural network computing. Each hidden layer neuron possesses its own nonlinear output function in a commonly utilized artificial neural network. It creates a one-to-one relationship between each nonlinear function and the hidden layer activation output to the next layer. Conversely, a dendritic neuron generates its activation output by aggregating outputs from multiple nonlinear functions, enabling it to form more efficacious synapses than a point neuron. This prompts the question: Does incorporating nonlinear dendrites into a neuron serve as an effective means of augmenting model capacity? To address this inquiry, we undertook a series of experiments.

In the first part of our experiment, our objective is to scrutinize the impact of amalgamating active dendritic outputs on the behavior of neural networks. Although incorporating dendrites into a neuron can theoretically bolster its information storage capacity, given that more synapses are available for storing information [13], whether this method is efficient persists. To address this, we draw comparisons between models composed of point neurons and those integrating dendritic structures of varying configurations.

### Dense model on ImageNet dataset

We begin by employing the Resnet-18 network [34] as a baseline point neuron-based model, a widely utilized computer vision model. In the case of models featuring active dendrites, we substitute point neurons in the original comparative models with dendritic neurons, as illustrated in Fig. 1E. We ensure that each nonlinear summation unit--whether it be a point neuron or an active dendrite unit--receives no more than one copy of input from the preceding network layer.

We first compare the baseline models with models where point neurons are directly substituted with dendritic neurons while maintaining the overall architecture. For this part of the experiment, each neuron's dendrite receives the same set of inputs from the last layer. The models were trained with widely used ImageNet dataset [35]. Further details about model training, evaluation and architecture can be found in the Methods section.

The results of this part of the experiment are displayed in the left half of Fig. 2. (A) shows the model training loss values, (B) illustrates the model accuracy on the training set, and (C) shows the test accuracy of the models. Each curve in the three panels compares models with the same inter-layer communication cost. The data points on the left end of each curve are from models comprised of point neurons, while the remaining data points are from models with dendritic neurons of different numbers of dendrites per neuron, indicated on the \(x\)-axis. We also scaled the baseline model by proportionally increasing or decreasing the number of channels per network layer, shown as different curves. In this way, the communication bandwidths between network layers were proportionally scaled.

The results shown in Fig. 2A and B indicate that adding dendrites can lead to an improved model expressivity, leading to better fittings to the training
data. When comparing models of the same inter-layer communication budget, the models with more dendrites consistently exhibit lower loss values and higher training accuracy than those with fewer or no dendrites.

Is it possible to translate the enhanced fitting capabilities conferred by dendritic neurons into tangible benefits, as measured by model test accuracy? As illustrated in Fig. 2C, incorporating additional dendrites into each neuron consistently results in improvements in test accuracy across all four distinct levels of inter-layer communication cost. It is evident that, under the same hidden layer communication channel width level, the integration of active dendrites can substantially enhance model capacity and performance, and the effect remains stable across different inter-layer communication width scales.

Enhancing model capacity while keeping a reasonable level of communication cost is a desirable goal. The increased number of parameters associated with additional dendrites can pose a significant challenge for networks that require a transfer of weights between on-chip and off-chip locations. Those dendrites' additional computing and space requirements can also be a big issue. To address this challenge, we investigated the effectiveness of replacing point neurons with dendritic neurons in computing cost, as measured by the total number of model parameters and FLOPs required for inference.

The outcomes of this experiment segment can be observed in the right half of Fig. 2. We compare three distinct levels of computational complexity to provide a clearer understanding of the results. Assume a dendritic neuron contains \(K\) branches, for the dense model we study here, each dendrite receives the same number of inputs/weights as a point neuron; that is, a dendritic neuron receives \(K\) times more inputs than a point neuron with the same input dimensionality. Assume that two sequential fully connected neural network layers both have \(D\) channels. For the second network layer, the computational complexity and the number of parameters will both be \(D^{2}\). Then for two dendritic neuron layers with \(\hat{D}\) channels, assume each neuron is equipped with \(K\) dendrites, the computational and parametric complexity will be \(K\hat{D}^{2}\). Therefore, for a model of dendritic neuron with \(K\) dendrites to have the same level of computational complexity as a point neuron-based model, we need to reduce the number of inter-layer communication channels in each network layer to \(1/\sqrt{K}\) of the original numbers, that is \(\hat{D}=D/\sqrt{K}\).

In Fig. 2D, E, and F, the blue dashed curves represent experimental results obtained from various models with standard complexity. The leftmost data point corresponds to the standard ResNet-18 model, which serves as a baseline for this group. Subsequent data points to the right denote dendritic models with \(K\) values of 4, 16, and 64, respectively. Concomitantly, these models' channels have been adjusted to 1/2, 1/4, and 1/8 of the original model's values, respectively, barring the input and output network layers. By maintaining this configuration, data points on the same curve exhibit equivalent parametric and computational complexities.

The orange curves demonstrate data from models in which the number of inter-layer communication channels has been uniformly scaled up by a factor
of two, while the green dashed curves represent models where the number of channels has been scaled by a factor of four. To facilitate a comprehensive understanding of the data, channel scale factors for each model, as compared to the standard model, have been explicitly labeled on the curves in Fig. 2F.

Our analysis yields a particularly intriguing result concerning the performance of dendritic neuron models compared to point neuron-based models under the constraints of equivalent computing and parametric complexity. As depicted in Fig. 2D, E, and F, we observe that a dendritic neuron model is capable of achieving comparable or superior fitting power relative to an equivalent point-neuron-based model when the channel width is set to be greater than or equal to one-fourth of the width of the baseline model. Remarkably, when the channel width is increased to half of the width of the baseline model, the dendritic neuron-based model consistently outperforms its point neuron-based counterpart regarding test accuracy.

### Further Findings

To further substantiate our research and enhance the robustness of our findings, we conducted supplementary experiments using a diverse array of model architectures and datasets. This analysis included an assortment of models encompassing those lacking residual connections, others that employed sparse network connections, as well as those leveraged transformer-based architectures. For the sake of clarity, we have included these additional results in the Appendix.

## 3 Communication cost analysis

The experimental analysis presented above has demonstrated that dendritic models can provide superior model capacity and performance while maintaining constrained inter-layer communication cost. To gain a better understanding on the benefits dendritic models can offer, we performed theoretical analysis on the full communication cost of point neuron and dendritic neuron-based models. Results reported in this part are based on the following considerations:

* We study and quantify the data movement process for computation between two sequential neural network layers, which can be easily generalized to many-layer settings.
* We evaluate the suitability of adopting dendritic networks for real-world applications, where wiring that follows a city-walk route is more relevant, by measuring the data movement path length using Manhattan distance metric.
* For the sake of clarity, a standard feed-forward network structure is employed for the analysis. However, the results obtained can also be applied to other network architectures.
* The computation of each network layer is executed within a discrete square area with dimensions of one-by-one.
* We only consider movement of neuron output values. This setting is most relevant for in-memory or near-memory computing. This is also relevant for computing with large batch size. No tiling based acceleration is considered.

We model the signal communication cost \(C_{T}\) as a sum of three parts: \(C_{A}\) is the cost for all processing elements (PEs) to propagate their outputs toward a convergence point (top-right corner is used) at the edge of the chip. \(C_{I}\) characterizes the process of transmitting data between two computational stages (network layers). \(C_{E}\) describes the communication cost associated with distributing signals within a chip for layer inference. This is represented by the following equation:

\[C_{T}=C_{A}+C_{I}+C_{E}\,. \tag{4}\]

### Cost with point neuron model

Our analysis initiates with a model predicated on point neurons. As previously mentioned, our investigation focuses on two network layers. We assume that the first layer sends an output of \(D\) dimensions to the second layer. For convenience and without loss of generality, we assume that each of the \(D\) dimensions originates from one PE on the chip.

In order to arrange \(D\) PEs on die area of size \(1\times 1\), each PE must have a height and width of \(l=1/\sqrt{D}\), resulting in an area size of \(1/D\). Similarly, the second layer is also composed of \(D\) PEs of the same size. Consequently, we obtain a grid of \(N\) by \(N\) PEs with \(N=\sqrt{D}\), with a distance of \(l\) between the center of each pair of neighboring PEs. See Fig. 11-A for a visual illustration.

For this arrangement we have

\[C_{A}=D(\sqrt{D}-1)l=D-\sqrt{D}\,, \tag{5}\]

as measured with Manhattan distance. Furthermore, an illustrative example of signal propagation within this context is provided in Fig. 11-B. Derivation of Eq. 5 can be found in Appendix. A.

\(C_{I}\) is very architecture dependent, thus we will abstain from attempting to estimate this component. We note \(C_{I}\) is linearly proportional to the size of \(D\). In scenarios where two network layers reside on different physical devices or the cost of moving data from PEs and memory is high, this portion of the cost may become the dominant communication expenses.

We assess \(C_{E}\) with minimal rectilinear spanning tree (MRST) algorithm [36]. Given a grid of \(N\times N\) PEs, the objective is to deliver every dimension of the data to each PE. The MRST algorithm enables us to determine the minimal path length required to connect all PEs, which is \((N^{2}-1)\cdot l\). An example path is illustrated in Fig. 11C. Consequently, we obtain the cost of delivering data as

\[C_{E}=(N^{2}-1)\cdot l\cdot D=(D-1)\sqrt{D}\,. \tag{6}\]
### Cost with dendritic neuron model

This section estimates the communication cost associated with a model based on dendritic neurons. To ensure a fair comparison, we have maintained the number of parameters and floating-point operations (FLOPs) consistent with those in the point neuron model scenario. Similar to the case of point neuron-based models, we also give the total cost \(\hat{C}_{T}\) in the following equation:

\[\hat{C}_{T}=\hat{C}_{A}+\hat{C}_{I}+\hat{C}_{E}\,. \tag{7}\]

Given that each neuron has \(K\) dendrites, one layer of the model under examination will have a total of \(M=D\sqrt{K}\) dendrites. As illustrated in Fig. 11-D, every group of \(K\) dendrites aggregates to form a single output dimension. Consequently, the first layer will produce an output with a dimensionality of \(\hat{D}=D/\sqrt{K}\), which serves to maintain an equivalent computational complexity as the point neuron-based model previously described. We reiterate our assumption that those \(\hat{D}\) neurons are arranged in a grid format, specifically of size \(\hat{N}\times\hat{N}\), with \(\hat{N}=\sqrt{\hat{D}}\).

We postulate that the computation of each dendrite is processed by one PE. In this scenario, the die area is divided into \(M\) units, with each unit occupying a specific area. The height and width of this area, denoted by \(\hat{l}\), can be calculated as \(\hat{l}=1/\sqrt{M}\). Through this, we arrive at the size of a PE for processing each dendrite being \(\frac{1}{D\sqrt{K}}\), which is \(1/\sqrt{K}\) of the point neuron-based model PE die size. This corresponds to the assumption that a dendrite in this analysis receives a proportion of \(1/\sqrt{K}\) of the inputs that a point neuron receives.

In light of the aforementioned derivation, we note that the signal transfer cost, denoted as \(\hat{C}_{A}\), consists of two components. The first component, \(\hat{C}_{AG}\), refers to the cost of aggregating dendritic outputs for each neuron. The second component, \(\hat{C}_{AA}\), represents the cost of transmitting the aggregated data of all neurons off the die. Their expressions are as follows.

\[\hat{C}_{AG} =(K-1)\cdot\hat{D}\cdot\hat{l}\] \[=\sqrt{D}(K^{1/4}-K^{-3/4})<\sqrt{D}K^{1/4}\,, \tag{8}\] \[\hat{C}_{AA} =\hat{N}\hat{N}(\hat{N}-1)\hat{l}(\sqrt{K})<\frac{D}{\sqrt{K}}\,,\] (9) \[\hat{C}_{A} =\hat{C}_{AG}+\hat{C}_{AA}<\sqrt{D}K^{1/4}+\frac{D}{\sqrt{K}}\,. \tag{10}\]

Akin to the point neuron models, we will not attempt to derive \(\hat{C}_{I}\), although we have the relationship of \(C_{I}=\sqrt{K}\cdot\hat{C}_{I}\) under the assumptions of the equivalent parameter/FLOPs count setting.

As for the \(\hat{C}_{E}\) component, note that the second layer receives \(\frac{D}{\sqrt{K}}\) inputs and consists of \(M\) units. Utilizing the MRST method, the cost associated with
one-dimensional input connecting to \(M\) units can be computed as \((M-1)\cdot\hat{l}\). We arrive at

\[\hat{C}_{E}=\frac{D}{\sqrt{K}}(D\sqrt{K}-1)\cdot\hat{l}\approx D^{\frac{3}{2}}/K ^{\frac{1}{4}}\,. \tag{11}\]

### Comparative analysis

From the above derivations, we are equipped to compare the communication costs associated with point neuron-based and dendritic neuron-based models under different configurations. As previously established, \(C_{I}\) and \(\hat{C}_{I}\) are highly dependent on the architecture, and \(\hat{C}_{I}\) is \(\sqrt{K}\) times smaller than \(C_{I}\). For the scope of this analysis, we will focus on analyzing \(C_{A}\), \(\hat{C}_{A}\), \(C_{E}\), and \(\hat{C}_{E}\).

Fig. 3A depicts the ratio of communication costs between the dendritic neuron-based model and the point neuron-based model. The results for different inter-layer dimensions \(D\) of the point neuron models and various channel reduction ratios \(\sqrt{K}\) are displayed within the figure. Notably, as \(\sqrt{K}\) increases, dendritic neuron-based models consistently demonstrate lower communication costs compared to their point neuron-based counterparts at the same level of computational complexity.

Upon examining \(\hat{C}_{A}\) and \(\hat{C}_{E}\), it becomes apparent that \(\hat{C}_{E}\) is typically much larger than \(\hat{C}_{A}\) when \(D\) assumes large values, a common scenario in this context. Moreover, in models characterized by sparse connectivity, \(\hat{C}_{A}\) remains unchanged regardless of connection sparsity levels, whereas \(\hat{C}_{E}\) can vary. As a result, it is necessary to delve deeper into the \(\hat{C}_{E}\) term and scrutinize its behavior within the context of sparse models.

In congruence with the approach adopted for dense models, we also employ the MRST algorithm to estimate the communication cost when dealing with sparse models. Considering the variability in the communication cost due to different sparse connection patterns, we sample a set of 100 random connection patterns for each setting to provide a robust estimate of the average cost.

Fig. 3B presents \(\hat{C}_{E}\) under varying model sparsity levels and diverse numbers of dendrites per neuron. Our observations reveal a negative power relationship between \(\hat{C}_{E}\) and \(K\), with a power of 0.51. This relationship is accurately mirrored in the \(K^{1/4}\) factor presented in simplified version of \(\hat{C}_{E}\) shown in Eq. 11.

## 4 Discussion

In this study, we were inspired by the observation that biological neurons aggregate the outputs of multiple dendrites to form a single output. Within each dendrite, the integration of synaptic inputs is nonlinear rather than a linear summation due to the presence of various voltage-gated ion channels. Accordingly, we have constructed neural network units that mimic these characteristics by integrating their synaptic inputs nonlinearly.

In the process of pooling outputs from multiple units, there is an inherent loss of information due to the many-to-one nature of the pooling function.
This suggests a potential decline in model performance. Such behavior is indeed noticeable when dealing with models that possess limited inter-layer bandwidth, that is those models that are equipped with a smaller number of channels. This observation holds when the models under comparison maintain the same parametric complexity. It is likely that this phenomenon led Goodfellow et al. [37] to observe suboptimal model performance when they utilized architectures that pool ReLU units.

However, our findings suggest that once the bandwidth is increased beyond a certain threshold, it becomes unnecessary to augment the model size by adding more channels. Instead, the addition of extra dendrites to neurons appears to offer superior efficiency in enhancing model performance.

The implications of this discovery are substantial for both theoretical perspectives and practical applications. Theoretically, it highlights that when we widen the architecture to expand models, we are actually augmenting the number of features within the hidden layers rather than enhancing the features propagated toward subsequent layers. This insight refines our understanding of the internal dynamics of neural network development and behavior.

Practically, the adoption of an active dendritic structure enables models to achieve superior performance compared to point neuron-based models, given a fixed inter-layer communication budget. This can lead to a linear reduction in memory access during neural network inference and a smaller memory footprint, particularly when large batch sizes are employed during model inference.

Figure 3: (A) Topographic representation of the ratio \((\hat{C}_{A}+\hat{C}_{E})/(C_{A}+C_{E})\): The visualization highlights the influence of the variations in the values of \(D\) and \(\sqrt{K}\) on the ratio \((\hat{C}_{A}+\hat{C}_{E})/(C_{A}+C_{E})\). The choice of \(\sqrt{K}\) over \(K\) was made to provide a clearer depiction of the relationship between the decrease in communication cost and the corresponding reduction in the layer’s output bandwidth. (B) Demonstrates the variations in \(\hat{C}_{E}\) in response to different quantities of dendrites per neuron, symbolized as \(K\), and varying levels of sparsity. The axes are portrayed in a logarithmic scale. When \(K=1\), the models are point neurons based. For this experiment, we have utilized a \(D\) value of 256.
Our comprehensive analysis of communication cost further reveals that adopting a dendritic structure can also yield a reduction in on-chip communication costs, following a square-root relationship to the inter-layer communication reduction ratio. Considering that communication costs dominate energy consumption in contemporary computing chips, our findings could significantly influence the design of future neural network accelerators.

Our findings shed light on critical insights; however, our comprehension of why channel sharing among a group of features can yield performance on par with, or even superior to, traditional models still needs to be more refined. In conventional models, each feature, or nonlinear neuron, establishes multiple connections directly to the succeeding network layer. One potential explanation posits that the pooling process in our dendritic layer can be viewed as a low-rank approximation of a significantly larger weight matrix \(W\in\mathbb{R}^{D\sqrt{K}\times D\sqrt{K}}\) using a smaller weight matrix \(W_{d}\in\mathbb{R}^{\frac{D}{\sqrt{K}}\times D\sqrt{K}}\). However, this interpretation provides only a limited perspective in comprehending this process. These gaps in our understanding necessitate further research to appreciate the dynamics and implications of such channel-sharing configurations fully.

Notably, the dendritic models used in our study are equipped with a single layer of nonlinearity, as opposed to two layers as suggested in Eq. 3. However, we observed improved performance in dendritic neuron models when adding an extra nonlinear function, particularly when neurons were equipped with many dendrites (results not shown). It would be intriguing to explore how different types of nonlinearity and more advanced nonlinear architectures would impact models.

Finally, our observation aligns with patterns observed in the evolution and development of the brain. In simpler, early-stage brains, neurons exhibit less structural complexity, consistent with the preference for point neuron-based models in smaller neural networks. However, as brains evolve to more advanced stages, neurons exhibit greater complexity and richer connectivity patterns, analogous to the preference for dendritic neuron-based models in larger neural network architectures [7]. This parallel suggests that incorporating dendritic neurons in artificial neural networks may reflect fundamental principles underlying the organization and functionality of biological neural systems. Our study contributes valuable insights into the comparative utility of dendritic and point neuron models in neural network design and offers guidance for their applications in various computational contexts.

## 5 Methods

### Datasets

The present study leverages three commonly used datasets: ImageNet, CIFAR-100, and LibriSpeech, for model training and evaluation. These datasets are commonly served as benchmarks in deep learning research.
**ImageNet Dataset:** For this study, we use the ILSVRC 2012 subset of the ImageNet dataset, which consists of 1.2 million training images and 50,000 validation images from 1,000 categories [35]. The images vary in size and are resized to a fixed resolution of 224x224 pixels for uniformity, per the standard ResNet procedure [34]. The typical data augmentation techniques, such as random cropping, random horizontal flipping, and color jittering, were applied during training to enhance the model's generalization ability.

**CIFAR-100 Dataset:** The dataset consists of 60,000 32x32 color images in 100 classes, with 600 images in each class. There are 50,000 training images and 10,000 test images [38]. Like the ImageNet data processing, we followed the typical data augmentation procedure [34].

**LibriSpeech dataset:** The dataset is a publicly available English speech corpus for Automatic Speech Recognition (ASR) training and evaluation from the LibriVox project's audiobooks. It consists of 1000 hours of transcribed speech, divided into training, development, and testing subsets [39]. The experiment utilizing this dataset can be found in Appendix B.

### Model architectures

In this study, we primarily used the ResNet-18 architecture as the baseline model. ResNet-18 is an 18-layer deep residual neural network, a seminal model proposed by He et al. [34]. The baseline configuration of ResNet-18 encapsulates an initial convolutional layer, followed by four residual blocks, each of which consists of two convolutional layers. This pattern constitutes the primary structure of our working model; in contrast to the original ResNet-18 model, our adapted architecture positions the shortcut connection after the ReLU (Rectified Linear Unit) activation function. This modification is imperative to ensure the compatibility of the dendritic structure with the model architecture.

For experiments on scaling up networks, we scaled up each network layer by the same designated factor except for the input and output of the model. For models with dendritic neurons, we replaced neurons in the standard model with dendritic neurons with \(K\) dendrites as specified by the experiment setting, except for the input and output layers of the model. To maintain the uniform model complexity scaling throughout the model, we equip the input layer and the penultimate layer of the model with neurons of \(\sqrt{K}\) instead of \(K\) dendrites. The same setting is also employed in experiments designed to compare models that share identical inter-layer communication costs.

For models trained on CIFAR-100, we observed training instability. Therefore we clipped the gradient norm to 1.0 during model training. We also added an extra batch norm to each dendrite to improve model stability. This additional batch norm can be fused with the previous layer and thus will not add extra computation burden at the inference stage.

In addition to models based on the ResNet-18 architecture, we have corroborated our findings using a model devoid of shortcut connections. This
strategy ensures that the benefits observed are not strictly confined to a particular architecture. The configuration of this model is delineated in Appendix B, where the corresponding experimental outcomes can also be found.

Moreover, our experimentation extended to the transformer-based model. Within this model, the standard feedforward layers are substituted with network layers based on dendritic neurons. Comprehensive details pertaining to this modification can be found in Appendix B.

### Model training

We trained all models with a cosine learning rate decay schedule and the SGD optimizer with a momentum of 0.9.

For ImageNet with dense ResNet models, the learning rate was initialized at 0.4 (instead of 0.1 to compensate for the batch size used for training), and models were trained for 120 epochs, including two warm-up epochs with a learning rate of 0.04. Weight decay was set to \(1\times 10^{-4}\). A batch size of 1024 was employed, and the training was distributed across 8 GPUs.

For ImageNet with sparse ResNet models, the models were trained for 200 epochs with an initial learning rate of 0.1 and 2 warm-up epochs at a learning rate of 0.01. The weight decay parameter was set to \(1\times 10^{-4}\). To achieve a sparse ratio of 85%, we applied L1-unstructured global pruning in 5 rounds, conducted between epochs 40 and 140. Subsequently, the models were trained for an additional 60 epochs.

Finally, for CIFAR-100 models, we trained them for 200 epochs with a learning rate of 0.05, including two warm-up epochs at a learning rate of 0.005. A batch size of 64 was utilized, and the weight decay parameter was set to \(5\times 10^{-4}\).

Our investigation emphasizes the comparative analysis of the performance of various models under identical training conditions, facilitating an equitable assessment of the distinct capabilities of each model. Consequently, all models within the comparison group undergo training with the same hyperparameters, barring the requisite architecture adjustments. Further details concerning the experiments can be found in the accompanying source code.

### Code availability

The entirety of the code used to produce the findings presented herein will be openly accessible to the public upon the publication of this paper.
