Faster and Accurate Neural Networks with Semantic Inference
Deep neural networks (DNN) usually come with a significant computational
burden. While approaches such as structured pruning and mobile-specific DNNs
have been proposed, they incur drastic accuracy loss. In this paper we leverage
the intrinsic redundancy in latent representations to reduce the computational
load with limited loss in performance. We show that semantically similar inputs
share many filters, especially in the earlier layers. Thus, semantically
similar classes can be clustered to create cluster-specific subgraphs. To this
end, we propose a new framework called Semantic Inference (SINF). In short,
SINF (i) identifies the semantic cluster the object belongs to using a small
additional classifier and (ii) executes the subgraph extracted from the base
DNN related to that semantic cluster for inference. To extract each
cluster-specific subgraph, we propose a new approach named Discriminative
Capability Score (DCS) that finds the subgraph with the capability to
discriminate among the members of a specific semantic cluster. DCS is
independent from SINF and can be applied to any DNN. We benchmark the
performance of DCS on the VGG16, VGG19, and ResNet50 DNNs trained on the
CIFAR100 dataset against 6 state-of-the-art pruning approaches. Our results
show that (i) SINF reduces the inference time of VGG19, VGG16, and ResNet50
respectively by up to 35%, 29% and 15% with only 0.17%, 3.75%, and 6.75%
accuracy loss (ii) DCS achieves respectively up to 3.65%, 4.25%, and 2.36%
better accuracy with VGG16, VGG19, and ResNet50 with respect to existing
discriminative scores (iii) when used as a pruning criterion, DCS achieves up
to 8.13% accuracy gain with 5.82% less parameters than the existing state of
the art work published at ICLR 2023 (iv) when considering per-cluster accuracy,
SINF performs on average 5.73%, 8.38% and 6.36% better than the base VGG16,
VGG19, and ResNet50.
Sazzad Sayyed, Jonathan Ashdown, Francesco Restuccia
# Faster and Accurate Neural Networks

###### Abstract

Deep neural networks (DNNs) usually come with a significant computational and data labeling burden. While approaches such as structured pruning and mobile-specific DNNs have been proposed, they incur in drastic accuracy loss. Conversely from prior work, in this paper we leverage the intrinsic redundancy in latent representations to drastically reduce the computational load with very limited loss in performance. Specifically, we show that semantically similar inputs share a significant number of filter activations, especially in the earlier layers. As such, semantically similar classes can be "clustered" so as to create cluster-specific subgraphs. These may be "turned on" when an input belonging to a semantic cluster is being presented to the DNN, while the rest of the DNN can be "turned off". To this end, we propose a new framework called _Semantic Inference_ (SINF). In short, SINF (i) identifies the semantic cluster the object belongs to using a small additional classifier; and then (ii) executes the subgraph extracted from the base DNN related to that semantic cluster to perform the inference. To extract each cluster-specific subgraph, we propose a new approach named _Discriminative Capability Score_ (DCS) that effectively finds the subgraph with the capability to discriminate among the members of a specific semantic cluster. Importantly, DCS is independent from SINF, as it is a general-purpose quantity that can be applied to any DNN. We benchmark the performance of DCS on the VGG16, VGG19, and ResNet50 DNNs trained on the CIFAR100 dataset against 6 state-of-the-art pruning approaches. Our results show that (i) SINF reduces the inference time of VGG19, VGG16, and ResNet50 respectively by up to 35%, 29% and 15% with only 0.17%, 3.75%, and 6.75% accuracy loss; (ii) DCS achieves respectively up to 3.65%, 4.25%, and 2.36% better accuracy with VGG16, VGG19, and ResNet50 with respect to existing discriminative scores; (iii) when used as a pruning criterion, DCS achieves up to 8.13% accuracy gain with 5.82% less parameters than the existing state of the art work published at ICLR 2023; (iv) when considering per-cluster accuracy, SINF performs on average 5.73%, 8.38% and 6.36% better than the base VGG16, VGG19, and ResNet50. We share our code for reproducibility.

## 1 Introduction

Deep neural networks (DNNs) have produced significant advances in computer vision (CV), [Krizhevsky et al.(2012); Kirillov et al.(2023); Redmon et al.(2016)], natural language processing (NLP) [Vaswani et al.(2017)], and multi-modal tasks [Radford et al.(2021)], just to name a few. Usually, DNNs process a very large number of parameters. For example, the state of the art YOLOv8 uses a DNN backbone with 53 layers and 40M parameters [Terven & Cordova-Esparza(2023)]. On the other hand, DNNs are increasingly being used in resource-constrained mobile systems. For example, unmanned autonomous vehicles (UAVs) and self-driving cars need to frequently perform object detection [Wu et al.(2020)] and semantic segmentation [Mo et al.(2022)] to avoid obstacles during navigation and build detailed 3D maps [Wang et al.(2020); Fraga-Lamas et al.(2019); Riti Dass (Medium) (2018)]. Applying state-of-the-art DNNs in these scenarios is hardly feasible for real-time edge deployment.

As discussed in Section 2, a plethora of existing work has been devoted to reduce the complexity of DNNs [Tan & Le(2019); Howard et al.(2017); Iandola et al.(2016); Sandler et al.(2018)]. Mobile
specific DNNs such as MobileNet [Sandler et al.(2018)] and MnasNet [Tan et al.(2019)] decrease the computational requirements at the detriment of accuracy. For example, MobileNet loses up to 6.4% in accuracy as compared to ResNet-152 [He et al.(2016)]. Alternative approaches include pruning [Han et al.(2015b); Li et al.(2018); Chen et al.(2019); Tanno et al.(2019); Singh et al.(2019); Kaya et al.(2019); Yao et al.(2017); Huang et al.(2020)], quantization [Han et al.(2015a); Qin et al.(2022); Cai et al.(2020)], and coding [Gajilala et al.(2020); Han et al.(2015a)], which also incur in excessive DNN performance loss. The key issue is that existing approaches do not guarantee faster inference as previous methods depend on the way the DNN is implemented in hardware. For example, unstructured pruning or weight pruning does not transfer to faster inference since the majority of the DNN circuitry has to be executed regardless of the application[Wen et al.(2016); Ma et al.(2022)]. On the other hand, structured pruning, which encompasses filter pruning and layer pruning, makes the inference faster and less expensive in terms of energy [Ma et al.(2022)].

In stark opposition with prior work, in this paper we propose _Semantic Inference_ (SINF) to reduce the number of weights for a DNN implemented on a edge device. Our key intuition is that in practical mobile settings, DNN inputs come only from a limited set of classes which are usually _semantically similar_ and are _highly correlated_ over time. For example, DNNs deployed in drone-based surveillance systems may only need to detect and identify certain classes (e.g., people, cars, animals) and will hardly encounter indoor objects. Another intuition - proven in Figure 1 - is that semantically similar inputs share a significant number of filter activations among themselves than semantically dissimilar inputs, especially in the earlier layers. For example, it is intuitive that images of seals share significantly more filter activations with images of dolphins than with images of tables. _We use this to logically rearrange the DNN, so that only the portion of the DNN that is devoted to classify the semantic relevant class that the current input belongs to will be executed_.

This paper makes the following novel contributions:

\(\bullet\) We propose a new inference framework called SINF, which logically partitions the DNN into subgraphs relevant to semantically similar classes, so that only the subgraph related to the cluster the object belongs to gets activated. To create the subgraphs, we propose a new Discriminative Capability Score (DCS) to find the filters that can best distinguish semantically similar classes;

\(\bullet\) We benchmark the performance of our SINF on VGG16, VGG19, and ResNet50 DNNs trained on the CIFAR100 dataset. We compare DCS against state-of-the-art discriminative algorithms by Mittal et al.(2019), Molchanov et al.(2019), Hu et al.(2016), Sui et al.(2021), and by Lin et al.(2020). We also use DCS as pruning approach and compare it against the work by Murti et al.(2023), which like DCS does not require retraining and/or fine-tuning. Our results show that (i) SINF reduces the inference time of VGG19, VGG16, and ResNet50 respectively by up to 35%, 29% and 15% with only 0.17%, 3.75%, and 6.75% accuracy loss; (ii) DCS achieves respectively up to 3.65%, 4.25%, and 2.36% better accuracy with VGG16, VGG19, and ResNet50 with respect to existing discriminative scores; (iii) when used as a pruning criterion, DCS achieves up to 8.13% accuracy gain with 5.82% less parameters than the existing state of the art work published at ICLR 2023; (iv) when considering per-cluster accuracy, SINF performs on average 5.73%, 8.38% and 6.36% better than VGG16, VGG19, and ResNet50.

## 2 Related Work

**Model Pruning**: The _lottery ticket hypothesis_ introduced by Frankle and Carbin(2019) has spurred a plethora of research work in DNN pruning. For example, Han et al.(2015b) proposed weight-norm-based unstructured pruning, while Paul et al.(2023) tries to explain the success of the magnitude-based pruning methods. Li et al.(2017) used the L\({}_{1}\) norm of the kernel weights to prune entire filters. However, weight-norm-based strategies do not directly take into account the importance of the filters or parameters to preserve the DNN accuracy. Another approach is first-order gradient based methods [Molchanov et al.(2017); Molchanov et al.(2019)] which estimate the importance of the filters based on the gradient of the loss function. Another class of techniques leverages the filter activation maps. For example, RoyChowdhury et al.(2017) study the presence of duplicate neurons and determines that convolutional layers are prone to developing redundant duplicate filters. To find such filters, Sui et al.(2021) uses the change in nuclear norm of the matrix formed from the activation maps when individual filters are removed from a layer. Lin et al.(2020) use the expected rank of the feature maps, while Chen et al.(2023) explain the soft-threshold pruning as an implicit case of Iterative Shrinkage-Thresholding. _Although these methods determine the redundant filters,
they fail to focus on the filters which are necessary to distinguish among the classes. Moreover, all of these methods require fine-tuning after pruning._ When the fine tuning is not possible, these methods do not provide satisfactory performance. Recently, Murti et al. (2023) propose a retrain-free _IterTVSPrune_ approach based on Total Variational Distance (TVD) Verdu (2014). The authors state there is no closed-form solution to TVD and approximate it with a simplified form of Hellinger distance to make it applicable to real-world DNNs. Here, we take a semantics-based approach and attempt to find filters able to best discriminate classes belonging to a given semantic cluster.

**Quantization and Coding**: Quantization and coding two methods complement other model compression and acceleration methods. The seminal work by Han et al. (2015) compressed the DNN through quantization and Huffman coding to reduce the memory footprint. Among more recent work, post-training quantization [Cai et al. (2020), Fang et al. (2020)] and quantization-aware training [Liu et al. (2021), Bhalgat et al. (2020), Zhong et al. (2022)] have been proposed. Qin et al. (2022) pushes the boundary using single-bit quantization of the popular language model Bidirectional Encoder Representations from Transformers(BERT). Li et al. (2020) designed a layer-wise symmetric quantizer with the learnable clip value only for high-level feature extraction module. Tu et al. (2023) recently designed an algorithm for network quantization catered to the needs of image super resolution. Gajjala et al. (2020) proposes three variants of Huffman encoding to compress the gradients for distributed training of neural networks. Both quantization and coding are complementary to the SINF and can be used to achieve further improvement in performance.

**Early Exit in Neural Network** Early exit was proposed by Teerapittayanon et al. to make the DNN inference dynamic by using auxiliary (and relatively small) neural networks attached to the output of the DNN layers. Based on the confidence of the prediction of the auxiliary networks, the decision to traverse the remaining layers is made [Matsubara et al. (2021)]. The training of the auxiliary classifiers can be done jointly with the backbone network as done by Elbayad et al. (2020), Zhou et al. (2020), and Pomponi et al. (2022). The classifiers can be trained using either cross-entropy loss [Lo et al. (2017), Wang et al. (2019)], or through knowledge distillation [Phuong & Lampert (2019), Li et al. (2019)]. The classifiers can also be trained separately as in Liu et al. (2020), Garg & Moschitti (2021), and Xin et al. (2020). Han et al. (2023) tries to improve the performance of the early classifiers by using block-dependent loss which uses information from a subset of the exits close to a block to train it. Dong et al. (2022) addresses the wasteful computation of the early auxiliary classifiers when they are not confident enough by predicting which early exit to use using a lightweight "Exit Predictor". Narayan et al. (2023) modeled the exit selection as an online learning problem and proposed to choose the exit in an unsupervised way. SINF uses an auxiliary classifier but the end goal is not early inference but to predict the next path to route the sample input to.

## 3 Dividing a DNN Into Semantic Subgraphs

Let \(\mathcal{D}\) be a labeled dataset with classes taken from a set \(\mathcal{K}\). We define a set of \(K\) clusters of classes \(\{\gamma_{1},\cdots,\gamma_{K}\}\), such that \(\gamma_{1}\cup\gamma_{2}\cup\cdots\gamma_{K}=\mathcal{K}\). We assume that these clusters are defined based on application-level similarities (e.g., classes related to flowers, insects, etc.) or pre-defined at the dataset level (e.g., as in the CIFAR100 dataset). We define \(\mathcal{F}\) as a DNN trained on dataset \(\mathcal{D}\). By viewing \(\mathcal{F}\) as a computation graph, we define the Semantic DNN Subgraph Problem (SDSP).

Synthetic DNN Subgraph Problem (SDSP)

Find \(K\) subgraphs \(\mathcal{F}_{\gamma_{i}}\cdots\mathcal{F}_{\gamma_{K}}\) such that

\[\mathcal{L}_{eval}(\mathcal{F},\mathcal{D})\leq\frac{1}{K}\sum_{i=1}^{K} \mathcal{L}_{eval}(\mathcal{F}_{\gamma_{i}},\mathcal{D}_{\gamma_{i}}), \tag{1}\]

where \(\mathcal{F}_{\gamma_{i}}\subset\mathcal{F}\) and \(\mathcal{D}_{\gamma_{i}}\subset\mathcal{D}\) are respectively the subgraphs of \(\mathcal{F}\) and subset of the dataset corresponding to the partition \(\gamma_{i}\). Score function \(\mathcal{L}_{eval}\) is the evaluation criterion being used to score \(\mathcal{F}\) on dataset \(\mathcal{D}\).

In other words, the subgraph \(\mathcal{F}_{\gamma_{i}}\) contains the nodes of \(\mathcal{F}\) which best classifies the members of the corresponding partition \(\gamma_{i}\) on \(\mathcal{D}_{\gamma_{i}}\). We perform a series of experiments to validate the intuition behind the SDSP. It is well known that DNN filters identify parts of objects, colors or concepts. These filters are shared among classes to reduce the number of parameters of the DNN [Bau et al. (2017)]. On the other hand, filter activations become sparser as the DNN becomes deeper, with filters reacting only to specific inputs belonging to specific classes. This phenomenon can be ob
served in the top portion of Figure 1, which shows the average filter activation strength for the "otter" and "seal" classes in the the 40th and 49th convolutional layers of ResNet50 trained on CIFAR100.

To obtain these results, we have taken the average of each filter activation vector for each input in the training set corresponding to the two classes, and performed min-max normalization to obtain values between 0 and 1. This experiment reinforces the notion that filters in earlier layers are less specialized than filters in deeper layers. Moreover, it remarks that filters from semantically similar classes get similarly activated, especially in earlier layers. _To put it in more quantitative terms, the \(L_{l}\) distance of the activation maps of the mentioned classes in the 40th layer is 0.028, while the same for the 49th layer is 0.111_. To further investigate this critical aspect, we have performed additional experiments where we have computed the percentage of filters "shared" among different classes for each layer of ResNet50. Specifically, for a filter in a given layer, we have tagged it with the top 20 classes it gets most activated for. For two classes, the sharing of the filters is calculated as the number of filters tagged with both classes over the number of filters tagged with at classes over the number of filters tagged with at the bottom portion of Figure 1, where the first row shows the filters shared between the "dolphin" and "wale" classes - two semantically similar classes - "dolphin" and "table". _As the numbers suggest, the semantically similar classes share more filters than semantically dissimilar classes_. These results further confirm that filter sharing among classes decreases as the DNN becomes deeper.

## 4 Discriminative Capability Score

We describe the procedure to obtain the DCS for a given layer \(l\) and given semantic cluster \(\gamma_{m}\) in Algorithm 1. Let us consider a generic layer \(l\) of a DNN with \(C^{l}_{out}\) number of output channels. We define the activation output of channel \(c_{i}\) of this layer for an input sample \(X^{j}\) as \(\mathbf{A}^{j}_{l,c_{i}}\), which has dimension \(C^{l}_{out}\times H\times W\) where \(H\) and \(W\) are respectively the height and width of the activation output. For each output channel, SINF performs an adaptive pooling operation \(\mathcal{P}(\cdot)\) to reduce the size of the feature map of each channel from \(H\times W\) to \(k\times k\). Flattening the feature map gives a feature vector \(\mathbf{F}^{j}_{c_{i}}\) of size \(k^{2}\). Next, the feature vectors obtained from all the filters are concatenated to generate the complete feature vector \(\mathbf{F}^{j}\), whose dimension is \(C^{l}_{out}\cdot k^{2}\) for layer \(l\). By defining \(N_{c}\) as the number of classes in the semantic cluster and \(N_{f}=C^{l}_{out}k^{2}\) as the number of features from the respective \(l\)-th layer of the DNN, DCS finds a transformation \(\mathbf{W}\in\mathbb{R}^{N_{c}\times N_{f}}\) that optimizes objective function \(\mathcal{L}_{DOF}\) as shown in Equation 2, where \(\mathbf{W}^{*}\) contains information about the discriminative capability of the filters of layer \(l\):

\[\mathbf{W}^{*}=\operatorname*{argmin}_{\mathbf{W}}\frac{1}{|\mathcal{D}_{ \gamma_{m}}|}\sum_{j=1}^{j=|\mathcal{D}_{\gamma_{m}}|}\mathcal{L}_{DOF}( \mathbf{W}\cdot\mathbf{F}^{j},t^{j}), \tag{2}\]

where \(t^{j}\) denotes the target value or label for the input sample \(X^{j}\) in the dataset \(\mathcal{D}_{\gamma_{m}}\) corresponding to a class cluster \(\gamma_{m}\). The weights in feature importance matrix \(\mathbf{I}=\mathbf{W}^{*}\odot\nabla_{\mathbf{W}^{*}}\mathcal{L}_{DOF}\) directly encode the importance of the individual features for distinguishing the classes. Here, \(\mathcal{L}_{DOF}\) is the objective function averaged over the samples. We define \(s_{i}\) as the norm of the \(i\)-th column of the feature importance matrix \(\mathbf{I}\). In other words, \(s_{i}\) represents the score contribution of a feature \(f_{i}\) in discriminating among the semantic classes. This converts the matrix \(\mathbf{I}\) into a vector \(s\) of length \(C^{l}_{out}\cdot k^{2}\). The final step is to attribute the DCS to individual filters. We use a group norm operation \(\mathcal{G}(\cdot)\) on the obtained feature score vector \(\mathbf{s}\) where the feature scores are grouped into vectors

Figure 1: (top) Filter activations in ResNet50; (bottom) Percentage of filters shared between (a) semantically similar classes – “dolphin” and “whale”; (b) semantically dissimilar classes – “dolphin” and “table”.
consisting of \(k^{2}\) consecutive feature score values. The DCS of channel \(c_{i}\) is obtained as follows:

\[\text{DCS}_{c_{i}}=\sqrt{\sum_{j}u_{c_{i,j}}^{2}} \tag{3}\]

where \(\text{DCS}_{c_{i}}\) is the desired discriminative capability score of the channel \(c_{i}\) and \(u_{c_{i},j}\) is the \(j\) th element of the feature cluster corresponding to the same filter.

We have investigated through experiments whether the DCS captures the phenomena explained in the previous section. Figure 2 shows the DCS distributions obtained in layer 6, 9, 11, and 13 of VGG16 by considering the cluster "fish" of CIFAR100. Figure 2 confirms that deeper layers are more specialized for individual classes, and thus the average DCS for the filters in the deeper layers is smaller - 0.68 for layer 6 vs 0.39 for layer 13. _In other words, in earlier layers more filters contribute to discriminate classes since filters are extracting more generalized features at early stages_. The decrease of average DCS implies that less filters are related to a given cluster.

## 5 Semantic Inference (SINF)

A step-by-step overview of the main operations of SINF is summarized in the top portion of Figure 3. Prior to the deployment of the DNN, SINF uses the DCS to construct the semantic subgraphs, as

Figure 2: The DCS distribution for cluster “fish” of CIFAR100 in different layers of VGG16.
explained later. After deployment, upon receiving an input, SINF first classifies which semantic cluster the input belongs to (**step 1**). To this end, a _Common Feature Extractor_ is trained to extract the features to correctly predict the semantic cluster (**step 2**). This is actually performed by the _Semantic Route Predictor_ (SRP), whose structure is detailed later in this section (**step 3**). Based on the SRP output, the input will be routed to the selected semantic subgraph by using a _Feature Router_ (**step 4**).

Finally, the inference output is obtained from the appropriate subgraph (**step 5**). We remark that although we are representing each subgraph separately for better graphical clarity, in practice the separation is only from a logical perspective. In other words, no additional memory beyond the annotations needed to characterize each subgraph is used by SINF.

**Extraction of Subgraphs.** The extraction of the subgraph follows the procedure described in Algorithm 2. We define \(L\) and \(M\) as respectively the last layer and the layer before the Common Feature Extractor. We define \(r_{X}\) as the percentage of retained filters in generic layer \(X\). For semantic cluster \(\gamma_{i}\), we iterate from layer \(L\) to layer \(M\) to extract the subgraph. For each layer \(M\leq l\leq L\), we calculate \(r_{l}\), as well as the DCS score of the filters using Algorithm 1. We rank the filters based on the score and the indices of the top \(r_{l}\) percent filters are saved. This is repeated for all the semantic clusters. If the average accuracy of the extracted subgraphs for the semantic clusters is above an accuracy threshold \(\tau_{acc}\), the indices of the filters belonging to the subgraphs are stored. This procedure is performed for different values of \(r_{L}\) and \(r_{M}\). In this paper, \(r_{L}\) is set between 90% and 10%, with steps of 10, while \(r_{M}\) is set between 10% and 1%, with steps of 2. The accuracy threshold \(\tau_{acc}\) is set to the accuracy of the original DNN as shown in Equation 1. Although we have used categorical cross entropy, this is DNN-independent and can be set to any objective function.

```
0: Partitioned Dataset \(\mathcal{D}=\{\mathcal{D}_{\gamma_{1}},\mathcal{D}_{\gamma_{2}}......\mathcal{D }_{\gamma_{K}}\}\)  Pretrained DNN: \(\mathcal{F}=\mathcal{F}_{L-1}\circ\mathcal{F}_{L-2}\circ\ldots\circ\mathcal{F }_{0}\)  Discriminative Objective Function: \(\mathcal{L}_{DOF}\)  Filter Retention Percentage at layer \(L\) = \(r_{L}\)  Filter Retention Percentage at layer \(M\) = \(r_{M}\)  Accuracy Threshold = \(\tau_{acc}\)
```
0: Filter annotations for the subgraphs for semantic clusters = \(\mathbf{SA}[]\)  SA = Empty dictionary() // To store the final filters annotations for extracted subgraphs for\(\mathcal{D}_{\gamma_{i}}\) in \(\mathcal{D}\)do  SA = Empty dictionary() for\(l=L,L-1,\ldots,M\)do \(r_{l}\gets r_{M}+\frac{(l-M)(r_{L}-r_{M})}{L-M}\) // Percentage of filters to retain at layer \(l\)  DCS\(l\gets DCS(\mathcal{F},\mathcal{D}_{\gamma_{i}},\mathcal{L}_{DOF})\) //Obtaining DCS Scores from algorithm 1  Rank the filters  Save the indices of top \(r_{l}\) percent filters in \(\mathbf{SA}_{\gamma_{i}}\) with layer number as key endfor  Calculate \(acc_{avg}=\frac{1}{K}\sum_{i=1}^{i=K}(accuracy(\mathcal{F}_{\gamma_{i}}))\) if\(acc_{avg}\geq\tau_{acc}\)then  Save \(\mathbf{SA}_{\gamma_{i}}\) in SA with \(\gamma_{i}\) as key endif endfor ```

**Algorithm 2** Subgraph Extraction for Semantic Clusters

**Semantic Route Predictor.** The purpose of the semantic route predictor is to predict the semantic clusters an input sample belongs to so that it can be forwarded towards the corresponding route for final prediction. An auxiliary classifier \(\boldsymbol{\chi}\), attached at the \(l\)-th layer of the base model \(\mathcal{F}\) is used for

Figure 3: Overview of Semantic Inference (SINF).
this task. To train the auxiliary classifier \(\mathbf{\chi}\), the section up to the \(l\)-th layer of \(\mathcal{F}\) is frozen and the activation map of this layer \(l\) is passed as input to the semantic route predictor. The earliest layer which provides good prediction (an accuracy threshold of 75% is used in this case) for semantic routes is chosen as the \(l\) th layer. The architecture of the auxilliary classifier consists of two convolution layers, followed by adaptive average pooling layer stacked on top of three fully connected layers. SINF uses the convolution layers to tailor the activation map from layer \(l\) of the base model for classification of the semantic clusters. The output of semantic route predictor is the probability distribution over the \(K\) different semantic clusters and the input is predicted to belong to the cluster with the highest probability. In our case, the value of \(K\) is 20 as we have 20 semantic clusters.

**Feature Router.** The DCS decision can be improved if SINF conditions the output of the semantic router predictor \(\mathbf{\chi}\) on its confidence. The feature router calculates this confidence and routes the activation maps to the appropriate specialized path. It takes the activation map from the semantic router predictor \(\mathbf{\chi}\) along with the probability distribution from its prediction layer. To compute the confidence of the classifier on individual decisions, the feature router employs the lightweight metric proposed by Park et al. (2015). The confidence score can be calculated as \(C_{\chi}=P_{h}-P_{sh}\), using the highest (\(P_{h}\)) and the second highest probabilities (\(P_{sh}\)) for individual semantic clusters. The confidence score is a proxy for the probability that the inference aligns with the correct label. The confidence score of a classifier correlates with the probability of correct inference as is shown by the Park et al. (2015). If the confidence score exceeds a threshold, the activation map is routed to the subgraph corresponding to the predicted semantic cluster - in other words, only the filters corresponding to this subgraph are turned on. Otherwise the base model is used for final decision.

## 6 Experimental Evaluation

To quantify improvement of SINF, we consider existing pruning approaches by Molchanov et al. (2019), Hu et al. (2016), Mittal et al. (2019), Sui et al. (2021), and Lin et al. (2020). In addition, to compare with a pruning approach that does not require retraining, we also consider the work Murti et al. (2023) published at ICLR 2023. We chose the CIFAR100 dataset for our experiments, since its 100 classes that are pre-grouped into 20 semantically similar classes.

Figure 4: (Top) Cumulative distribution of the confidence values of SINF for the VGG16, VGG19, and ResNet50 DNNs. (Bottom) Accuracy, vs relative inference time and confidence threshold.
**Impact of Confidence Threshold.** We first evaluate the impact of the confidence threshold \(\gamma\), given its importance in achieving the right balance between latency and accuracy in SINF. The top row of Figure 4 shows the cumulative distribution of the confidence values of SINF for the VGG16, VGG19, and ResNet50 DNNs computed over the test set. We can observe that the confidence distribution increases in skewness and average, which reflects the increasingly better predictive capabilities of the examined DNNs. In other words, the confidence score of SINF will be higher in a DNN exhibiting better accuracy. In addition, the bottom portion of Figure 4 shows the decrease in accuracy and the relative latency with respect to the original DNN as a function of the confidence threshold. As expected, increasing the confidence score increases the accuracy while also decreasing the gain in latency. As such, the confidence threshold \(\gamma\) can be used as hyper parameter to find the needed trade-off between accuracy and latency. We notice that with VGG19, the overall accuracy actually increases by up to 0.49% for confidence threshold values greater than 0.4 and smaller than 0.9. If we look at the confidence distribution of VGG19, we can see that the auxiliary classifier is more confident about its predictions as compared to the case of VGG16. For VGG19, we can get up to 41.8% reduction in inference time for only 2.83% drop in accuracy. In the best case, SINF reduces the inference time by up to 35%, 29% and 15% with only 0.17%, 3.75%, and 6.75% accuracy loss for VGG19, VGG16, and ResNet50 respectively.

**DCS vs Existing Discriminative Metrics.** To evaluate the effectiveness of DCS with respect to prior approaches, we use existing discriminative metrics proposed in existing work while keeping the same inference structure of SINF. Figure 5 compares DCS against gradient-based approaches _Sensitivity_ by Mittal et al. (2019) and _Taylor_ by Molchanov et al. (2019), an approach based on sparsity of activation named _APOZ_ by Hu et al. (2016), an approach based on channel-independence named _CHIP_ by Sui et al. (2021), and an approach based on channel importance named _HRANK_ by Lin et al. (2020). All the approaches are compared without retraining the DNN, as proposed in SINF. Figure 5 shows that in the best case, DCS has 15% higher accuracy than the second-best approach _Taylor_ for VGG16 with 75% sparsity (i.e., percentage of parameters dropped). For VGG19, DCS achieves in the best case 6.54% higher accuracy than the second-best approach _Taylor_ at 63% sparsity. Lastly, in the case of ResNet50, the best case is attained at 51% sparsity, where DCS presents 14.87% more accuracy than the second-best approach _Sensitivity_. On average, SINF achieves 3.65%, 4.25%, and 2.36% better accuracy than the second-best approaches for VGG16, VGG19, and ResNet50 respectively.

**DCS as Pruning Criterion.** Since DCS scores the capability of each filter to distinguish a given class, DCS may be used as a pruning criterion. In this case, the dataset can be viewed as a single macro-cluster and DCS can be applied to determine the most relevant filters. For comparison, we consider the state-of-the-art _IterTVSPrune_ by Murti et al. (2023) published at ICLR 2023, which also does not require fine-tuning. _IterTVSPrune_ performs iterative search with a threshold on the maximum accuracy loss to find the smallest DNN satisfying the accuracy constraint. For fair comparison, we have taken the percentage of parameters pruned by _IterTVSPrune_ at each layer and set the same pruning threshold for DCS. In other words, we prune Table 1 summarizes the performance achieved by _DCS_ and _IterTVSPrune_. We did not compare on CIFAR100 with ResNet50 as the authors of Murti et al. (2023) did not provide the performance obtained by applying their approach on ResNet50 trained with CIFAR100. We notice that for different DNN structures and datasets, DCS achieves substantially better performance in 3 out of 5 settings considered while

Figure 5: Comparison between DCS and state of the art for VGG19, VGG16, and ResNet50.
achieving similar performance in the remaining 2 settings. The best results are obtained in the case of VGG16 trained on CIFAR 100 and ResNet50 trained on CIFAR10, where we see respectively 9.75% and 8.13% accuracy gain with 3.6% and 5.82% less parameters.

**Per-Cluster Accuracy Gain.** We posed ourselves the following question: "_Can SINF perform better than the original DNN when considering the accuracy obtained in individual clusters?_" To answer this question, we performed the following experiment. When applying DCS to find the cluster-specific subgraphs, we inserted a constraint that the resulting dataset-level accuracy should be at least as much as the original DNN. Then, we find the subgraphs the with lowest percentage of parameters retained. Figure 6 shows the accuracy gain obtained on the individual clusters as compared to the original VGG16, VGG19, and ResNet50 DNNs. Intriguingly, _SINF provides on the average 5.73%, 8.38% and 6.36% better per-cluster accuracy than the original VGG16, VGG19, and ResNet50 DNNs, respectively, notwithstanding that the number of parameters being used are 30%, 50%, and 44% less than the original DNNs._ We believe the reason behind this improvement is that the semantic partitioning performed by SINF improves the DNN by making it more explainable. This way, the DNN becomes "less confused" among different semantic clusters, which justifies better results when considering per-cluster accuracy.

## 7 Concluding Remarks

In this paper, we have proposed a new framework called _Semantic Inference_ (SINF) which allows to achieve faster execution of DNNs without compromising accuracy. As part of SINF, we have proposed a new approach named _Discriminative Capability Score_ (DCS) to find subgraphs inside large DNNs to discriminate among the members of a specific semantic cluster. We have benchmarked the

\begin{table}
\begin{tabular}{|c c c c c c|} \hline
**DNN** & **Dataset** & **Pruned** & **Criterion** & **Accuracy Loss** & **Difference** \\ \hline VGG16 & CIFAR100 & 40.2\% & IterTVSPrune & 18.59\% & +9.75\% Accuracy \\  & & 43.8\% & DCS & 8.84\% & -3.6\% Parameters \\ \hline VGG16 & CIFAR10 & 37.6\% & IterTVSPrune & 1.9\% & +0.61\% Accuracy \\  & & 42\% & DCS & 1.29\% & -4.4\% Parameters \\ \hline VGG19 & CIFAR100 & 59\% & IterTVSPrune & 5.2\% & +0.05\% Accuracy \\  & & 59\% & DCS & 5.15\% & +0\% Parameters \\ \hline VGG19 & CIFAR10 & 49\% & IterTVSPrune & 1.3\% & +0.4\% Accuracy \\  & & 49.65\% & DCS & 0.9\% & -0.65\% Parameters \\ \hline ResNet50 & CIFAR10 & 34.1\% & IterTVSPrune & 9.94\% & +8.13\% Accuracy \\  & & 39.92\% & DCS & 1.81\% & -5.82\% Parameters \\ \hline \end{tabular}
\end{table}
Table 1: Using DCS as a Pruning Criterion vs _IterTVSPrune_ (ICLR 2023).

Figure 6: Performance gain compared to the original DNN. The x-axis shows the ids of different semantic clusters from CIFAR100 dataset and y-axis shows the performance improvement when models corresponding to specific classes are used.
performance of SINF on the VGG16, VGG19 and ResNet50 DNNs trained on the CIFAR100 and CIFAR10 datasets. By comparing the performance of SINF with several existing approaches, we have shown that SINF outperforms the state of the art.
