To prune or not to prune : A chaos-causality approach to principled
  pruning of dense neural networks
Reducing the size of a neural network (pruning) by removing weights without
impacting its performance is an important problem for resource-constrained
devices. In the past, pruning was typically accomplished by ranking or
penalizing weights based on criteria like magnitude and removing low-ranked
weights before retraining the remaining ones. Pruning strategies may also
involve removing neurons from the network in order to achieve the desired
reduction in network size. We formulate pruning as an optimization problem with
the objective of minimizing misclassifications by selecting specific weights.
To accomplish this, we have introduced the concept of chaos in learning
(Lyapunov exponents) via weight updates and exploiting causality to identify
the causal weights responsible for misclassification. Such a pruned network
maintains the original performance and retains feature explainability.
Rajan Sahu, Shivam Chadha, Nithin Nagaraj, Archana Mathur, Snehanshu Saha
# To Prune or not to Prune: :

###### Abstract

Reducing the size of a neural network (pruning) by removing weights without impacting its performance is an important problem for resource constrained devices. In the past, pruning was typically accomplished by ranking or penalizing weights based on criteria like magnitude and removing low-ranked weights before retraining the remaining ones. Pruning strategies may also involve removing neurons from the network in order to achieve the desired reduction in network size. We formulate pruning as an optimization problem with the objective of minimizing misclassifications by selecting specific weights. To accomplish this, we have introduced the concept of chaos in learning (Lyapunov exponents) via weight updates and exploiting causality to identify the causal weights responsible for misclassification. Such a pruned network maintains the original performance and retains feature explainability

## 1 Introduction

Designing a neural network architecture is a critical aspect of developing neural networks for various AI tasks, particularly in deep learning. One of the fundamental challenges in designing neural networks is finding the right balance between model complexity and sample size, which can have a significant impact on the network's performance. In general, a larger network with more parameters (overparameterized) can potentially learn more complex functions and patterns from the data [21]. However, challenge in network architecture design is to find the right balance between model complexity and sample size, so that the network can learn to generalize well to new, unseen data. In this context, over-parameterized networks [27], [19] have become increasingly popular in the deep learning era due to
their ability to achieve high expressivity and potentially better generalization performance [24]. The idea behind over-parameterization is to increase the number of parameters in the network beyond what is strictly necessary to fit the training data and remarkable generalization to test data. However, such networks still require a number of assumptions and hyperparameter tuning to guarantee optimal performance.

Pruning techniques should reduce the number of parameters in a neural network without compromising its accuracy. However, it is important to ponder why we do not simply train a smaller network from scratch to make training more efficient. The reason is that the architectures obtained after pruning are typically more challenging to train from scratch [16], and they often result in lower accuracy compared to the original networks. Therefore, while standard pruning techniques can effectively reduce the size and energy consumption of a network, it does not necessarily lead to a more efficient training process.

### Problem statement and Contributions

We pose a broad _Research Question_ here: Is there a principled approach to pruning overparameterized, dense neural networks to a reasonably good sparse approximation such that performance is not compromised and explainability is retained? It is well known that dense neural network training and particularly weight updates via SGD have some element of chaos [26],[9]. We expect that, between the successive weight updates due to SGD and miss-classification, there is some observed causality and non-causal weights (parameters) can be pruned, leading to a sparse network i.e. some weight updates cause reduction in network (training) loss and some do not! Can we train a dense network till a few epochs to derive a pruned architecture for the derivative to run for the remaining epochs and produce performance metrics in the \(\epsilon-\)ball of the original, dense network? Does this sparse network also train well, verified with Shapley [17] and WeightWatcher (_WW_) [18] tests.? Specifically, we contribute to the following:

* Present a unique and unifying framework on chaos and causality for deep network pruning. The unifying framework uses LE [10] and Granger causality (GC) [6] tandem.
* Propose novel pruning architectures, Lyapunov Exponent Granger Causality driven Fully Trained Network (LEGCNet-FT) and Lyapunov Exponent Granger Causality driven Partially Trained Network (LEGCNet-PT).
* LEGCNet-FT and LEGCNet-PT compare very well in performance and other baselines, Random [15] and Magnitude based [14] pruning techniques.
* Establish feature consistency of LEGCNet-FT and LEGCNet-PT in explainability.
* Verify empirically that the proposed architectures for pruning are not over-trained and obviously not over-parameterized but are still able to generalize well, on diverse data sets while saving Flops (FLOATing-point OPerationS). We accomplish this via the _WW_ test.

In summary, we propose pruning techniques, _LEGCNet-FT and LEGCNet-PT_ which perform at par with the dense, unpruned architecture and the existing pruning baselines. _While maintaining consistent performance, these techniques also help reduce epochs to converge and Flops to compute while maintaining feature consistency with their dense counterparts and ensuring proper training across layers validated via WW statistics._

## 2 Technical Motivation

Chaos, causality and the manifestation of the Lottery Ticket Hypothesis are the key motivation behind our proposed pruning mechanism.

### Gradient Descent and Low Dimensional Chaos

Is the process of updating weights in backpropagation via Gradient Descent chaotic? Is there an alternative interpretation of the minima in the weight landscape via low-dimensional chaos? The weight update in SGD is written as \(w_{i+1}\gets w_{i}-\eta_{i}\nabla_{w}f(w_{i})\) may be thought of as a discretization to the first order ODE: \(w^{\prime}(t)=-\nabla_{w}f(w_{i})\). The minimizer of the SGD is therefore conceived as a stable equilibrium of the ODE. That is, the minimum, \(w^{\star}\) can be thought of as a fixed point to the iterates \(w_{i+1}=G(\eta_{i},w_{i})\equiv w^{\star}=G(\eta_{i},w^{\star})\).

**Empirical evidence of chaos in back propagation:** We performed a series of experiments on different datasets to check Sensitive Dependence On Initial Conditions (SDIC) for weight initialization, on single hidden layer neural network. The weight initialization matrix \(W_{ij}\) followed Gaussian distribution \((W_{ij}\sim N(0,\sigma^{2}))\). We recorded two set of executions - one with initial weight \(w_{11}:w_{ij}\forall i\in 1..h,\forall j\in 1..n\), where \(n,h\) are the number of input and hidden neurons - and another, with infinitesimal perturbation (\(w_{11}+\delta\)) keeping other parameters same. Each time, the network
was trained via gradient descent and weight series were recorded. The method was repeated for the second weight connection \(w_{ij},i=1,j=2\) and for all the weights on the network. Later, the Lyapunov exponents were computed by using the TISEAN package [8] on the recorded weight series to measure the perturbed trajectory due to initial perturbation \(\delta\). We observed positive Lyapunov exponents which marked the presence of some chaotic behavior in gradient descent.

### Chaos and Causality

One way to address the issue of explainability in AI/machine learning is to seek causal explanations for choices made in the learning process. Conversely, a learning process that incorporates choices made out of causal considerations is easier to explain and interpret. This is the motivation behind using causality based criteria for the choice of what to prune (or not prune) in this study. To this end, we employ Granger Causality (GC) [7].

The principle of pruning that is causally-informed is formulated as follows. Those connections (weights) in the learning network that do not causally impact the loss are chosen for pruning. To determine the causal impact of a particular connection to the loss, we perform GC between the windowed Lyapunov exponents (LE) of the weight time series for that connection and the classification accuracy. The rationale behind this is the intuition that the chaotic signature of weight updates inform learning. A biological inspiration for chaotic signatures as a marker for learning is the empirical fact that neurons in the human brain exhibit chaos [4; 11] at all spatio-temporal scales. Starting from single neurons to coupled neurons to network of neurons to different areas of brain - chaos has been found to be ubiquitous to brain [11]. Chaotic systems are known to exhibit a wide range of patterns (periodic, quasi-periodic and non-periodic behaviors), are very robust to noise and enable efficient information transmission [20], processing/computation [3; 12] and classification [2]. There is also some evidence to suggest that weak chaos is likely to aid learning [25]. Thus our choice of testing causal strength between LE (a value \(>0\) is a marker of chaos) and classification accuracy as a criterion for pruning to yield sparse subnetworks that capture the learning of the task at hand.

### Lottery ticket hypothesis

The "lottery ticket hypothesis" [5] is a concept in neural network pruning that suggests that within a dense and over-parameterized neural network, there exist sparse sub-networks that can be trained to perform just as well as the original dense network. Any fully-connected feed-forward network \(f^{d}(x;\phi)\), with initial parameters \(\phi\) when trained on a training set D, \(f^{d}\) achieves a test accuracy \(a\) and error \(e\) at iteration \(j\). Our work LEGCNet, validates the lottery ticket hypothesis by finding the "winning-ticket", \(m\), to construct the sparse network, \(f^{s}\) such that \(acc^{s}\geq a\) and \(j^{s}<j\) where \(\|\phi\|\gg\|m\|\).

The remainder of the paper is organized to present the key methodologies used to develop the pruning technique (section 3), followed by a detailed experimental setup (section 4) and strong empirical evidence of the proposed technique in contrast to the baselines (section 5,6).

## 3 Tools and Methodology

### Definitions:

* Let \(f^{d}\) be a dense neural network of depth \(l\) and width \(h\) defined as \[f^{d}(x)=W_{i}^{d}\sigma_{i}(W_{i-1}^{d}\sigma_{i}(...W_{1}^{d}(x)))\] (1) where \(W_{i}^{d}\) is the weight matrix for layer \(i\) such that \(i\in 1..l\).
* Let \(f^{s}\) be sparse neural network of the same architecture as \(f^{d}\), with depth \(l\) and width \(h\).
* In order to validate the working of LEGCNet, we divided the method into two discrete approaches. In one approach, the entire training weight-series is used for computing Lyapunov exponents, testing Granger causality and for the identification of causal weights. Essentially, the dense network is trained till convergence and the approach is called LEGCNet-Full Train (LEGCNet-FT). In the second approach, LEGCNet-PT, the network is trained only till certain epochs (10% of the total iterates) and these few weight-updates are captured for identifying the causal weights.

### WeightWatcher (\(Ww\)) as a diagnostic tool

\(WW\) is a powerful open-source diagnostic tool designed for analyzing Deep Neural Networks (DNN). \(WW\) analyzes each layer by plotting the empirical spectral distribution (ESD), which represents the histogram of eigenvalues from the
layer's correlation matrix. Additionally, it fits the tail of the ESD to a (truncated) power law and presents these fitted distributions on separate axes. This visualization approach provides a clear representation of the eigenvalue distribution and highlights the presence of heavy-tailed behavior in the network's layers. In general, the ESDs observed in the best layers of high-performing DNNs can often be effectively modeled using a Power Law (PL) function. The PL exponents, denoted as alpha, tend to be closer to 2.0 in these cases indicating a heavy-tailed behavior in the layer's correlation matrix.

### SHAP as a diagnostic tool

SHAP (SHapley Additive exPlanations) is a game-theoretic technique utilized to provide explanations for the output of machine learning models. SHAP enables a comprehensive understanding of the contributions made by different features in the model's output, facilitating insightful explanations for its decision-making process. If a network is pruned according to some underlying principles, then the consistency in feature explainability is maintained before and after pruning i.e. the features which explain the outcome before pruning (fully connected, dense network) remain consistent on the pruned network.

### Windowed Weight Updates:

Consider a neural network with \(n\) inputs and \(r\) outputs, \(l\) hidden layers of \(h\) neurons, and, the input vector denoted as \(x\in R^{n}\), The network when trained by SGD generates a sequence of weight updates represented by \(w^{ji}=\left[w_{0}^{ji},w_{1}^{ji},...,w_{k}^{ji},...,w_{K}^{ji}\right]\) where \(w_{k}^{ji}\) is the weight of ith neuron of the input layer and jth neuron of the hidden layer at kth iteration. Considering, the weights being collected for initial few epochs, the weight iterates for hidden layer and output layer are \(W_{h}=\left\{w^{ji}\right\},W_{o}=\left\{w^{kj}\right\}\quad\forall i\in\left\{ 1,..,n\right\},\forall j\in\left\{1,..,m\right\},\forall k\in\left\{1,..,r\right\}\) An infinitesimal perturbation \(\delta_{0}\) is introduced in the initial weight \(w^{11}\), given as \(w^{11\delta_{0}}=w^{11}+\delta_{0}\), keeping other parameters - weights (initialization), learning rate, optimizer, and loss function- same. The network is then retrained with the perturbed weight, and the weights updates are recorded again as \(W_{h}^{\delta_{0}}=\left\{w^{ji\delta_{0}}\right\},W_{o}^{\delta_{0}}=\left\{ w^{kj\delta_{0}}\right\}\quad\forall i\in\left\{1,..,n\right\},\forall j\in\left\{1,..,m \right\},\forall k\in\left\{1,..,r\right\}\) A difference series obtained by subtracting perturbed weights from initial weights is \(\delta W_{h}=\left\{\delta w^{ji}\right\},\delta W_{o}=\left\{\delta w^{kj} \right\}\quad\forall i\in\left\{1,..,n\right\},\forall j\in\left\{1,..,m \right\},\forall k\in\left\{1,..,r\right\}\). We divide the weight series \(\delta w^{ji}\) into \(K\) windows, \(w^{ji}=\bigcup_{l=1}^{K}w^{ji(l)}\), and compute the Lyapunov exponent of all the windowed-weight trajectories. The series of the Lyapunov exponent \(\lambda\) of the windowed-weight trajectories \(w^{ji(K)}\) are represented using the notation \(\left\{\lambda^{ji\left\{1\right\}},\lambda^{ji\left\{2\right\}},...,\lambda^{ ji\left\{K\right\}}\right\}\). Additionally, we record the accuracy at every window during training \(\forall l\in 1,\ldots,K\), captured for weights at \(w^{ji(l)}\) and \(w^{kj(l)}\forall i\in\left\{1,..,n\right\},\forall j\in\left\{1,..,m\right\}, \forall k\in\left\{1,..,r\right\}\).

### Approximation capability of LEGCNet

The main theorem and lemma in this section build that the LEGCNet pruned network (\(f^{s}\)) is \(\epsilon\)-close to \(f^{d}\) with probability 1-\(\delta\)[22]. Consider \(f^{d}\) to be a dense neural network defined as

\[f^{d}(x)=W_{i}^{d}\sigma_{i}(W_{i-1}^{d}\sigma_{i}(...W_{1}^{d}(x))) \tag{2}\]

Figure 1: Weights pruned via LEGCNet method for selecting connections in sparse neural network
where \(W_{i}^{d}\) is the weight matrix for layer \(i\) such that \(i\in 1..l\) and \(h_{i}>h_{0},h_{i}>h_{l},\forall i\in 1...(l-1)\). We assume that for network in (2), \(\sigma_{i}\) is \(L_{i}\)-Lipschitz and weight matrix \(W_{i}^{d}\) is initialized from uniform distribution \(U[\frac{-K}{\sqrt{max(h_{i},h_{i-1})}},\frac{K}{\sqrt{max(h_{i},h_{i-1})}}]\), for some constant \(K\).

**Theorem 3.1** (_Approximation Capability of the Sparse Network)_: _Let \(\epsilon>0\), \(\delta>0\), \(\alpha\in(0,1)\) such that the, for some constants \(K_{1},K_{2},K_{3},K_{4},K_{5}\),_

\[h\geqslant max\left\{K_{1}^{\frac{1}{n}},\left(\frac{K_{2}}{\epsilon}\right) ^{\frac{1}{\alpha}},\left(\frac{K_{3}}{\delta}\right)^{\frac{1}{\alpha}},K_{ 4}+K_{5}log\left(\frac{1}{\delta}\right)\right\}\]

_then sparse network \(f^{s}\) obtained from LEGCNet by the mask \(m\), and pruning the weights \(W_{i}^{d}\), \(\forall i\in 1...l\) is \(\epsilon\)-close to \(f^{d}\), with the probability (\(1-\delta\)), i.e._

\[\sup_{x\in B_{d0}}\left\|f^{s}(x)-f^{d}(x)\right\|_{2}\leq\epsilon\]

**Remark:** Lipschitz property of the activation functions [23] is a necessary condition to validate the approximation capability of the proposed sparse network. We have used _sigmoid activation_ in the sparsely trained/pruned network.

**Lemma 3.2**: _Sigmoid activation is Lipschitz._

**Proof:** If a function \(f(x)\) is Lipschitz continuous, then: \(\left\|f(x)-f(y)\right\|\leq K\left\|x-y\right\|\equiv\left\|f^{\prime}(x) \right\|\leq K\). If \(K<1,f\) is a contraction map as well. We know that Sigmoid, \(\sigma(x)=\frac{1}{1+e^{-x}}\); and \(\left\|\sigma^{\prime}\right\|=\left\|\sigma(x)(1-\sigma(x))\right\|\). It's easy to follow that : \(\left\|\sigma(x)(1-\sigma(x))\right\|\leq\left\|\sigma(x)\right\|\left\|1- \sigma(x)\right\|\leq C_{1},C_{2}\). Since 0\(\leq C_{1}\), \(C_{2}\leq\) 1, \(C_{1}*C_{2}=\delta\leq 1\). Hence sigmoid is Lipschitz.

### Methodology Overview (refer figure 1)

Our pruning method was developed as follows. Initially, we trained a simple Multi-layer Perceptron (MLP) on a given dataset. Throughout the training process, we recorded the weights at each iteration, resulting in a time series of weight values. These weight time series were subsequently utilized to estimate the Lyapunov exponents, a measure of chaotic behavior, using the TISEAN package in conjunction with MATLAB scripts. This estimation was performed using a sliding window approach, generating a time series of Lyapunov exponents.

Based on our experiments, the time series of Lyapunov exponents consistently exhibited positive values, indicating the presence of chaotic elements in the weight time series. Consequently, our study focused on understanding whether these weights Granger caused the model's accuracy. Any weights that did not demonstrate this causal relationship were pruned before conducting subsequent model runs (code is uploaded in supplementary, zipped file).

## 4 Experimental set-up

In our study, we employed Python3.10 and Matlab R2022a to conduct experiments on a single hidden layer neural network on various datasets. Our experiments were conducted on Ryzen 9 3900XT Desktop Processor with 32GB RAM and 1TB HDD. During training, we stored the weight updates for every connection in CSV files. We assumed a window size of 200 iterates and computed the Lyapunov exponent for each weight connection on every window. Further, we calculated the training and test accuracy on every window, to capture the misclassification rates. Thus, we obtained a sequence of windowed-Lyapunov exponents and windowed accuracies for every connection. We then computed the Granger causality between the windowed Lyapunov exponents and the misclassification rate to identify weight connections that Granger caused misclassification. In the process of pruning the network, we removed the connections for which the Lyapunov exponents were found to Granger cause misclassification. After pruning, we reran the experiment by keeping the initial weights, optimizers, and other hyperparameters the same as unpruned network. We extended the work on different datasets and recorded the epochs and accuracies of the pruned network. Interestingly, we observed that the accuracies of the sparse network exceeded those of the dense network.

In our study, we conducted experiments in two parts. In the first part, we trained the neural network until convergence (LEGCNet-FT) and computed windowed-Lyapunov exponents for each weight connection as well as windowed accuracies. We then computed the Granger causality and pruned the network by removing connections that were found to cause misclassification. In the second part of the experiment, we trained the network only for few epochs (LEGCNet-PT) and used these initial weight updates to compute windowed-Lyapunov exponents and accuracies, repeating the same procedure as in the first part of the experiment. Finally, we compared the performance of the pruned network (LEGCNet-FT and LEGCNet-PT) to that of the original network. The results of all these experiments were recorded and presented in tables. Code is available as supplementary file.
## 5 Performance comparison - dense network, LEGCNet-FT, and LEGCNet-PT

We ran the experiments on seven tabular datasets - Cancer, Titanic, Banknote, Iris, Iris (3 features) Vowel and MNIST. The datasets were divided into 80:20 train-test split and the code was run 5 times, each time maintaining different network initialization. The best results from every initialization are reported in tables 1 and 2. We compared the Flops, % pruned (fraction of parameters removed *100), accuracy, f1-scores, and epochs for all methods (dense, LEGCNet-FT and LEGCNet-PT). Table 1 shows the performance comparison of dense network and LEGCNet-FT. Remarkably, LEGCNet-FT achieves notable reductions in Flops without compromising accuracy. Furthermore, LEGCNet-FT converges significantly faster consuming a few epochs compared to the dense network. Specifically, for the Titanic, Vowel, and Cancer datasets, LEGCNet-FT achieves convergence in just half the number of epochs required by the dense network. Nonetheless, both network achieves a similar level of performance - accuracy, and f1-score- without significant differences, thus validating the lottery ticket hypothesis. Table 2 demonstrates the performance of LEGCNet-PT. It shows that LEGCNet-PT performs at par with its counterpart, in terms of convergence speed, Flops, accuracy and F1 scores.

### Diagnostics

It is also crucial to examine the impact of our pruning technique on the training process of the model and determine if the relevance of feature importance is maintained and if the pruned network is properly trained or not. To accomplish this goal, we utilized two diagnostic tools, _namely WW and SHAP_. The _WW_ validates the compliance of our architecture, as reflected in the table 4 and figure 2 plotted for MNIST. The alpha lies between 2.0 and 6.0 on every layer (table 4). The ESD plots of the three types of training (dense, LEGCNet-PT, LEGCNet-FT) manifest a heavy-tailed distribution of eigenvalues on each layer indicating the layers are well-trained (figure 2). A careful observation at figure 2 reveals the following: ESD plot of a layer, where the orange spike on the far right is the tell-tale clue; it's called a Correlation Trap [13]. A Correlation Trap refers to a situation where the empirical spectral distributions (ESDs) of the actual (green) and random (red) distributions appear remarkably similar, with the exception of a small correlation shelf located just to the right of 0. In the random ESD (red), the largest eigenvalue (orange) is noticeably positioned further to the right

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|c|} \hline Dataset (hidden neurons) & Flops - & Flops- & Non causal & Epochs & Epochs & Accuracy & Accuracy & Fl1-score & F1-score & \#LFPuned \\  & DN & LEGCNet-PT & Weights & DN & LEGCNet-FT & DN & LEGCNet-FT & DN & LEGCNet-FT & LEGCNet-FT \\ \hline cancer(s) & 60 & 54 & 6 & 0 & 38 & 0.8579 & 0.8866 & 0.8060 & 0.8570 & 10 \\ \hline Titanic (8) & 56 & 52 & 4 & 43 & 20 & 0.7225 & 0.7177 & 0.6918 & 0.6843 & 7.14 \\ \hline BasalRec (8) & 40 & 36 & 4 & 9 & 7 & 0.9018 & 0.8899 & 0.9016 & 0.8906 & 10 \\ \hline Iris (6) & 42 & 40 & 2 & 182 & 166 & 0.9 & 0.9 & 0.9124 & 0.5419 & 4.76 \\ \hline Iris(7) features(3) & 36 & 26 & 10 & 139 & 125 & 0.9 & 0.9 & 0.9333 & 0.904 & 27.78 \\ \hline Word (4) & 36 & 34 & 2 & 36 & 14 & 0.7462 & 0.7538 & 0.7877 & 0.74 & 5.36 \\ \hline Miami (50, 30) & 41000 & 40861 & 139 & 27 & 30 & 0.9121 & 0.9165 & 0.8609 & 0.8781 & 0.34 \\ \hline \end{tabular}
\end{table}
Table 1: Comparing the performance of sparse network with dense networks by using LEGCNet across different datasets. The approach used for finding non-causal weights is LEGCNet-Fully Trained. Accuracies used are on the Test set. Dense Network - DN, Sparse Network - SN

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|} \hline Data (\(Epochs^{*}\)) & Flops (SN) & Epochs (SN) & Epochs (SN) & Accuracy (SN) & Accuracy (SN) & F1-score (SN) & F1-score (SN) & f1-score (SN) \\  & Random & Random & Magnitude & Random & Magnitude \\ \hline Cancer & 42 & 0.8759 & 0.8055 & 0.8660 & 0.8814 \\ \hline Titanic & 35 & 0.7201 & 0.7531 & 0.8842 & 0.6906 \\ \hline BasalRec & 6 & 0.8945 & 0.9618 & 0.8943 & 0.9015 \\ \hline Iris & 179 & 0.9000 & 0.9000 & 0.9124 & 0.9124 \\ \hline Iris\(3\) features & 160 & 0.9000 & 0.9313 & 0.9310 & 0.9330 \\ \hline Visual & 28 & 0.7642 & 0.7662 & 0.7229 & 0.7407 \\ \hline MNIST & 31 & 0.9119 & 0.9312 & 0.8627 & 0.8866 \\ \hline \end{tabular}
\end{table}
Table 3: Results for Random and Magnitude-based pruning;
and is separated from the majority of the ESD's bulk. This phenomenon indicates the presence of strong correlations in the layer, which can potentially affect the overall behavior and performance of the network. This is the case of a well-trained layer.

_SHAP:_ The SHAP values computed for the proposed pruning architecture as well as for the baselines - random and magnitude - indicate that the feature importance is maintained in LEGCNet-PT and LEGCNet-FT when compared with dense (figure 3). However, the baseline pruning methods (random, magnitude pruning) could not maintain the feature consistency as seen in the SHAP plots (supplementary file, section C). Though magnitude pruning shows feature consistency for Cancer, banknote, and Titanic datasets, the random pruning could not.

## 6 Discussion and Conclusion

Unlike the current baselines, the percentage of pruned weights is significantly less. This is because only the non-causal weights are pruned, weights that play no role in impacting the loss/accuracy. We argue that the proposed strategy is efficient and accurate, with the additional benefit of passing network fitting and explainability tests, in addition to satisfying the lottery ticket hypothesis (tables 1 and 2 Experimental section.). One of the other salient features of LEGCNet is that, unlike other pruning methods, it does not need the dense network to be trained for the entire cycle of epochs to identify pruning candidates. Rather, such candidates are detected after a few initial epochs so that the retraining can start immediately. This reflects in reduced Flops without compromising key performance indicators (tables 1, 2). Notably, our architecture is validated for correct training via \(WW\) Statistics as detailed in the diagnostic section previously.

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|c|} \hline  & \multicolumn{2}{c|}{Layer1: 784-50} & \multicolumn{2}{c|}{Layer2: 50-30} & \multicolumn{2}{c|}{Layer3: 30-10} \\ \hline Model & alpha & alpha, w & alpha & alpha, w & alpha & alpha, w \\ \hline Dense & 2.19 & 1.63 &.51 &.28 & 1.94 & 2.76 \\ \hline LEGCNet-FT & 2.24 & 1.64 & 1.51 & 1.83 & 2.29 & 3.20 \\ \hline LEGCNet-PT & 2.19 & 1.71 & 1.51 & 1.85 & 2.73 & 3.87 \\ \hline Random & 2.30 & 1.53 & 1.70 & 1.95 & 2.00 & 2.84 \\ \hline Magnitude & 2.19 & 1.63 & 1.55 & 1.85 & 1.96 & 2.78 \\ \hline \end{tabular}
\end{table}
Table 4: Weight Watcher summary retrieved for all models trained on the same initialization for MNIST

Figure 3: Shap values and feature importance computed on Cancer dataset (Banknote and Titanic plots can be seen in the supplementary file section D) for all the three models - Dense Network, LEGCNet-FT and LEGCNet-PT; the feature importance for the dense network is same as LEGCNet-FT and LEGCNet-PT

Figure 2: \(WW\) **plots for dense and LEGCNet-FT, and LEGCNet-PT networks on MNIST data (layer 1)**. Layers 2 and 3 are kept in the supplementary file (section A). Plots reveal the correct training of the proposed architectures. WW plots of MNIST for random and magnitude pruning are in supplementary file section B.
The experimental results demonstrated that the proposed pruning method exhibits notable advantages in maintaining feature importance compared to the traditional random and magnitude pruning methods. The feature consistency remained relatively stable after employing the proposed pruning technique, which was not the case for the other two methods. In the case of random and magnitude pruning, significant fluctuations in feature importance were evident after pruning. These fluctuations could potentially hinder the interpretability of the underlying model. However, our proposed pruning method demonstrated remarkable resilience in preserving feature importance, with minimal perturbations observed in SHAP plot patterns enabling a more interpretable and transparent pruned model. For mission-critical tasks on edge devices such as predicting power consumption of applications [1] or forecasting real-time blood glucose prediction, feature explainability on pruned networks are critical as they help determine accurate prediction when dimensionality is a curse. It is crucial to emphasize that our primary focus in this investigation was on the effects of pruning, rather than achieving perfect classification performance. We compared three distinct pruning techniques: random pruning, magnitude pruning, and our pruning approach. Overall, the experimental outcomes validate the superiority of the proposed pruning method. The findings hold great promise for further advancements in network optimization and model explainability. Our pruning approach is yet to be tested on baseline architectures (Resnet, Densenet), and Large Language Models and savings in carbon emission needs to be computed.

## Acknowledgments

Archana Mathur and Nithin Nagaraj would like to thank SERB-TARE (TAR/2021/000206) for supporting the work. Snehanshu Saha would like to thank the DBT-Builder project, Govt. of India (BT/INF/22/SP42543/2021) and SERB-SURE, DST.
