Learning to Linearize Deep Neural Networks for Secure and Efficient
  Private Inference
The large number of ReLU non-linearity operations in existing deep neural
networks makes them ill-suited for latency-efficient private inference (PI).
Existing techniques to reduce ReLU operations often involve manual effort and
sacrifice significant accuracy. In this paper, we first present a novel measure
of non-linearity layers' ReLU sensitivity, enabling mitigation of the
time-consuming manual efforts in identifying the same. Based on this
sensitivity, we then present SENet, a three-stage training method that for a
given ReLU budget, automatically assigns per-layer ReLU counts, decides the
ReLU locations for each layer's activation map, and trains a model with
significantly fewer ReLUs to potentially yield latency and communication
efficient PI. Experimental evaluations with multiple models on various datasets
show SENet's superior performance both in terms of reduced ReLUs and improved
classification accuracy compared to existing alternatives. In particular, SENet
can yield models that require up to ~2x fewer ReLUs while yielding similar
accuracy. For a similar ReLU budget SENet can yield models with ~2.32% improved
classification accuracy, evaluated on CIFAR-100.
Souvik Kundu, Shunlin Lu, Yuke Zhang, Jacqueline Liu, Peter A. Beerel
# Learning to Linearize Deep Neural Networks

###### Abstract

The large number of ReLU non-linearity operations in existing deep neural networks makes them ill-suited for latency-efficient private inference (PI). Existing techniques to reduce ReLU operations often involve manual effort and sacrifice significant accuracy. In this paper, we first present a novel measure of non-linearity layers' ReLU sensitivity, enabling mitigation of the time-consuming manual efforts in identifying the same. Based on this sensitivity, we then present SENet, a three-stage training method that for a given ReLU budget, automatically assigns per-layer ReLU counts, decides the ReLU locations for each layer's activation map, and trains a model with significantly fewer ReLUs to potentially yield latency and communication efficient PI. Experimental evaluations with multiple models on various datasets show SENet's superior performance both in terms of reduced ReLUs and improved classification accuracy compared to existing alternatives. In particular, SENet can yield models that require up to \(\sim\)\(2\times\) fewer ReLUs while yielding similar accuracy. For a similar ReLU budget SENet can yield models with \(\sim\)\(2.32\%\) improved classification accuracy, evaluated on CIFAR-100.

## 1 Introduction

With the recent proliferation of several AI-driven client-server applications including image analysis (Litjens et al., 2017), object detection, speech recognition (Hinton et al., 2012), and voice assistance services, the demand for machine learning inference as a service (MLaaS) has grown.

Simultaneously, the emergence of privacy concerns from both the users and model developers has made _private inference_ (PI) an important aspect of MLaaS. In PI the service provider retains the proprietary models in the cloud where the inference is performed on the client's encrypted data (_ciphertexts_), thus preserving both model (Kundu et al., 2021) and data-privacy (Yin et al., 2020).

Existing PI methods rely on various cryptographic protocols, including homomorphic encryption (HE) (Brakerski & Vaikuntanathan, 2014; Gentry, 2009) and additive secret sharing (ASS) (Goldreich et al., 2019) for the linear operations in the convolutional and fully connected (FC) layers. For example, popular methods like Gazelle (Juvekar et al., 2018), DELPHI (Mishra et al., 2020), and Cheetah (Reagen et al., 2021) use HE while MiniONN (Liu et al., 2017) and CryptoNAS (Ghodsi et al., 2020) use ASS. For performing the non-linear ReLU operations, the PI methods generally use Yao's Garbled Circuits (GC) (Yao, 1986). However, GCs demand orders of magnitude higher latency and communication than the PI of linear operations, making latency-efficient PI an exceedingly difficult task. In contrast, standard inference latency is dominated by the linear operations (Kundu et al., 2021) and is significantly lower than that of PI.

This has motivated the unique problem of reducing the number of ReLU non-linearity operations to reduce the communication and latency overhead of PI. In particular, recent literature has leveraged

Figure 1: Comparison of various methods in accuracy vs. #ReLU trade-off plot. SENet outperforms the existing approaches with an accuracy improvement of up to \(\sim\)\(4.5\%\) for similar ReLU budget.
neural architecture search (NAS) to optimize both the number and placement of ReLUs (Ghodsi et al., 2020; Cho et al., 2021). However, these methods often cost significant accuracy drop, particularly when the ReLU budget is low. For example, with a ReLU budget of 86k, CryptoNAS costs \(\sim\)\(9\%\) accuracy compared to the model with all ReLUs (AR) present. DeepReDuce (Jha et al., 2021) used a careful multi-stage optimization and provided reduced accuracy drop of \(\sim\)\(3\%\) at similar ReLU budgets. However, DeepReDuce heavily relies on manual effort for the precise removal of ReLU layers, making this strategy exceedingly difficult, particularly, for models with many layers. A portion of these accuracy drops can be attributed to the fact that these approaches are constrained to remove ReLUs at a higher granularity of layers and channels rather than at the pixel level. Only very recently, (Cho et al., 2022) proposed \(l_{1}\)-regularized pixel level ReLU reduction. However, such approaches are extremely hyperparameter sensitive and often do not guarantee meeting a specific ReLU budget. Moreover, the large number of training iterations required for improved accuracy may not be suitable for compute-limited servers (Mishra et al., 2020).

**Our contributions.** Our contribution is three-fold. We first empirically demonstrate the relation between a layer's sensitivity towards pruning and its associated ReLU sensitivity. Based on our observations, we introduce an automated layer-wise ReLU sensitivity evaluation strategy and propose SENet, a three-stage training process to yield secure and efficient networks for PI that guarantees meeting a target ReLU budget without any hyperparameter-dependent iterative training. In particular, for a given global ReLU budget, we first determine a sensitivity-driven layer-wise non-linearity (ReLU) unit budget. Given this budget, we then present a layer-wise ReLU allocation mask search. For each layer, we evaluate a binary mask tensor with the size of the corresponding activation map for which a 1 or 0 signifies the presence or absence of a ReLU unit, respectively. Finally, we use the trained mask to create a partial ReLU (PR) model with ReLU present only at fixed parts of the non-linearity layers, and fine-tune it via distillation from an iso-architecture trained AR model. Importantly, we support ReLU mask allocation both at the granularity of individual pixels and activation channels.

To further reduce both linear and non-linear (ReLU) layer compute costs, we extend our approach to SENet++. SENet++ uses a single training loop to train a model of different channel dropout rates (DRs) \(d_{r}\) (\(d_{r}\leq 1.0\)) of the weight tensor, where each \(d_{r}\) yields a sub-model with a MAC-ReLU budget smaller than or same as that of the original one. In particular, inspired by the idea of ordered dropout (Horvath et al., 2021), we train a PR model with multiple dropout rates (Horvath et al., 2021), where each dropout rate corresponds to a scaled channel sub-model having number of channels per layer \(\propto\) the \(d_{r}\). This essentially allows the server to yield multiple sub-models for different compute requirements that too via a single training loop, without costly memory footprint. Table 1 compares the important characteristics of our methods with existing alternatives.

We conduct extensive experiments and ablations on various models including variants of ResNet, Wide Residual Networks, and VGG on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet datasets. Experimental results show that SENet can yield SOTA accuracy-ReLU trade-off with an improved accuracy of up to \(\sim\)\(2.32\%\) for similar ReLU budgets. SENet++ (\(d_{r}=0.5\)) can further improve the MAC and ReLU cost of SENet, with an additional saving of \(4\times\) and \(\sim\)\(2\times\), respectively.

## 2 Preliminaries and Related Work

**Cryptographic primitives.** We briefly describe the relevant cryptographic primitives in this section.

_Additive secret sharing._ Given an element \(x\), an ASS of \(x\) is the pair \((\langle x\rangle_{1},\langle x\rangle_{2})=(x-r,r)\), where \(r\) is a random element and \(x=\langle x\rangle_{1}+\langle x\rangle_{2}\). Since \(r\) is random, the value \(x\) cannot be revealed by a single share, so that the value \(x\) is hidden.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline
**Name** & **Method** & **Reduced** & **Generativity** & **Before mode** & **Leporporato dynamic** \\  & **used** & **non-linearity** & **dimension** & **channel dropping** \\ \hline \multicolumn{4}{c|}{**Imagenet pruning**} & Vavonen & \(\beta\) & SENet weight & \(\beta\) & \(\beta\) \\ \hline Structural pruning & Vavonen & \(\gamma\) & SENet, Slice & \(\gamma\) & \(\beta\) \\ \hline Sohnma (Cho et al., 2021) & NAS & \(\gamma\) & Lucky Stack & \(\beta\) & \(\beta\) \\ \hline CycleGAN (Choi et al., 2020) & NAS & \(\gamma\) & Lucky Stack & \(\beta\) & \(\beta\) \\ \hline DBLPHPH (Mishra et al., 2020) & NAS + PA & \(\gamma\) & Linearback & \(\beta\) & \(\beta\) \\ \hline SENet (Lou et al., 2021) & NAS + PA & \(\gamma\) & Quantied & \(\beta\) & \(\beta\) \\ \hline DeepReDuce (Cho et al., 2021) & Manual + PA & \(\gamma\) & Lucky Stack & \(\beta\) & \(\beta\) \\ \hline SNL (An et al., 2022) & \(l_{1}\)-regularized & Manual, PASC & \(\gamma\) & Linear, PASC & \(\beta\) \\ \hline SNL(An et al., 2022) & \(l_{2}\)-regularized & Manual, PASC & \(\gamma\) & Linear, PASC & \(\beta\) \\ \hline SENet (ours) & Automated & \(\gamma\) & Quantied, PASC & \(\gamma\) & \(\beta\) \\ \hline SENet++ (ours) & Automated & \(\gamma\) & Quantied, PASC & \(\gamma\) & \(\beta\) \\ \hline \end{tabular}
\end{table}
Table 1: Comparison between existing approaches in yielding efficient models to perform PI. Note, SENet++ can yield a model that can be switched to sub-models of reduced channel sizes.
_Homomorphic encryption._ HE (Gentry, 2009) is a public key encryption scheme that supports homomorphic operations on the ciphertexts. Here, encryption function \(E\) generates the ciphertext \(t\) of a plaintext \(m\) where \(t=E(m,pk)\), and a decryption function \(D\) obtains the plaintext \(m\) via \(m=D(t,sk)\), where \(pk\) and \(sk\) are corresponding public and secret key, respectively. In PI, the results of linear operations can be obtained homomorphically through \(m_{1}\)\(\circ\)\(m_{2}=D(t_{1}\star t_{2},sk)\), where \(\circ\) represents a linear operation, \(\star\) is its corresponding homomorphic operation, \(t_{1}\) and \(t_{2}\) are the ciphertexts of \(m_{1}\) and \(m_{2}\), respectively.

_Garbled circuits._ GC (Yao, 1986) allows two parties to jointly compute a Boolean function \(f\) over their private inputs without revealing their inputs to each other. The Boolean function \(f\) is represented as a Boolean circuit \(C\). Here, a garbler creates an encoded Boolean circuit \(\tilde{C}\) and a set of input-correspondent labels through a procedure \(Garble(C)\) to send \(\tilde{C}\) and the labels to the other party who acts as an evaluator. The evaluator further sends the output label upon evaluation via \(Eval(\tilde{C})\). Finally, the garbler decrypts the labels to get the plain results to share with the evaluator.

**Private inference.** Similar to (Mishra et al., 2020), in this paper, we focus on a semi-honest client-server PI scenario where a client, holding private data, intends to use inference service from a server having a private model. Specifically, the semi-honest parties strictly follow the protocol but try to reveal their collaborator's private data by inspecting the information they received. On the other hand, a malicious client could deviate from the protocol.

To defend against various threats existing cryptographic protocols (Mishra et al., 2020; Ghodsi et al., 2020; Lou et al., 2021) rely on the popular online-offline topology (Mishra et al., 2020), where the client data independent component is pre-computed in the offline phase (Juvekar et al., 2018; Mishra et al., 2020; Ghodsi et al., 2021; Lehmkuhl et al., 2021). For the linear operations, DELPHI (Mishra et al., 2020) and MiniONN (Liu et al., 2017) move the heavy primitives in HE and ASS to offline enabling fast linear operations during PI online stage. However, the compute-heavy \(Eval(\tilde{C})\) of GC keeps the ReLU cost high even at the online stage.

**ReLU reduction for efficient PI.** Existing works use model designing with reduced ReLU counts via either search for efficient models (Mishra et al., 2020; Ghodsi et al., 2020; Lou et al., 2021; Cho et al., 2021) or manual re-design from an existing model (Jha et al., 2021). In particular, SAFENet (Lou et al., 2021) enables more fine-grained channel-wise substitution and mixed-precision activation approximation. CryptoNAS (Ghodsi et al., 2020) re-designs the neural architectures through evolutionary NAS techniques to minimize ReLU operations. Sphynx (Cho et al., 2021) further improves the search by leveraging differentiable macro-search NAS (Liu et al., 2018a) in yielding efficient PI models. DeepReDuce (Jha et al., 2021), on the other hand, reduced ReLU models via a manual effort of finding and dropping redundant ReLU layers starting from an existing standard model. Finally, a recent work (Cho et al., 2022) leveraged \(l_{1}\)-regularization to remove ReLU at the pixel level to yield SOTA accuracy vs non-linearity trade-off. However, the extreme hyperparameter dependence of such methods often provide sub-optimal solution and does not necessarily guarantee meeting a target ReLU budget. Moreover, a resource-limited server (Mishra et al., 2020) may not afford costly iterative training (Cho et al., 2022) in reducing the ReLU count.

## 3 Motivational Study: Relation between ReLU importance and Pruning Sensitivity

Existing work to find the importance of a ReLU layer (Jha et al., 2021), requires manual effort and is extremely time consuming. In contrast, model pruning literature (Lee et al., 2018; Kundu et al., 2021) leveraged various metrics to efficiently identify a layer's sensitivity towards a target pruning ratio. In particular, a layer's _pruning sensitivity_ can be quantitatively defined as the accuracy reduction caused by pruning a certain ratio of parameters from it (Ding et al., 2019). In particular, recent literature leveraged sparse learning (Ding et al., 2019; Kundu et al., 2021) and used a trained sparse model to evaluate the sensitivity of a layer \(l\) (\(\eta_{\theta^{l}}\)) as the ratio \(\frac{\texttt{total}\ \texttt{of non-zero layer parameters}}{\texttt{total}\ \texttt{ if layer parameters}}\).

**Despite significant progress in weight pruning sensitivity, due to the absence of any trainable parameter in the ReLU layer, its sensitivity for a given ReLU budget is yet to be explored.** We hypothesize that there may be a correlation between a layer's pruning sensitivity (Kundu et al., 2021) and the importance of ReLU and have conducted the following experiments to explore this.

Let us assume an \(L\)-layer DNN model \(\Phi\) parameterized by \(\mathbf{\Theta}\in\mathbb{R}^{m}\) that learns a function \(f_{\Phi}\), where \(m\) represents the total number of model parameters.
The goal of DNN parameter pruning is to identify and remove the unimportant parameters from a DNN and yield a reduced parameter model that has comparable performance to the baseline unpruned model. As part of the pruning process for a given parameter density \(d\), each parameter is associated with an auxiliary indicator variable \(c\) belonging to a mask tensor \(\mathbf{c}\in\{0,1\}^{m}\) such that only those \(\theta\) remain non-zero whose corresponding \(c=1\). With these notations, we can formulate the training optimization as

\[\text{min }\mathcal{L}(f_{\Phi}(\mathbf{\Theta}\odot\mathbf{c}))\text{, s.t. }\|\mathbf{c}\|_{0}\leq d\times m \tag{1}\]

where \(\mathcal{L}(.)\) represents the loss function, which for image classification tasks is generally the cross-entropy (CE) loss. We used a sparse learning framework (Kundu et al., 2021) to train a ResNet18 on CIFAR-100 for a target \(d=0.1\) and computed the pruning sensitivity of each layer. In particular, as shown in Fig. 2, earlier layers have higher pruning sensitivity than later ones. This means that to achieve close to baseline performance, the model trains later layers' parameters towards zero more than those of earlier layers.

We then compared this trend with that of the importance of different ReLU layers as defined in (Jha et al., 2021). In particular, we first identified five different modules of ReLU placement in a ResNet18, the pre-basic-block (BB) stem, BB1, BB2, BB3, and BB4. We then created five ResNet18 variants with ReLU non-linearity present only at one of the modules while replacing non-linearity at the other modules with identity layers. We identify the modules yielding higher accuracy to be the ones with higher ReLU importance (Jha et al., 2021). We then normalized the importance of a ReLU stage with accuracy Acc as the ratio \((\texttt{Acc}-\texttt{Acc}_{min})/(\texttt{Acc}_{max}-\texttt{Acc}_{min})\). Here Acc\({}_{max}\) and Acc\({}_{min}\) correspond to the accuracy of models with all and no ReLUs, respectively.

As depicted in Figure 2, the results show that the ReLU importance and parameter pruning sensitivity of a layer are inversely correlated. This inverse correlation may imply that a pruned layer can afford to have more zero-valued weights when the associated ReLU layer forces most of the computed activation values to zero.

## 4 SENet Training Methodology

As highlighted earlier, for a large number of ReLU layers \(L_{r}\), the manual evaluation and analysis of the candidate architectures become inefficient and time consuming. Moreover, the manual assignment of ReLU at the pixel level becomes even more intractable because the number of pixels that must be considered, explodes. To that end, we now present SENet, a three-stage automated ReLU trimming strategy that can yield models for a given reduced ReLU budget.

### Sensitivity Analysis

Inspired by our observations in Section 3, we define the ReLU sensitivity of a layer \(l\) as

\[\eta_{\mathbf{\alpha}^{i}}=(1-\eta_{\mathbf{\theta}^{i}}) \tag{2}\]

It is important to emphasize that, unlike ReLU importance, ReLU sensitivity does not require training many candidate models. However, \(\eta_{\mathbf{\theta}^{i}}\) can only be evaluated for a specific \(d\). We empirically observe that \(d>0.3\) tends to yield uniform sensitivity across layers due to a large parameter budget. In contrast, ultra-low density \(d<0.1\), costs non-negligible accuracy drops (Liu et al., 2018; Kundu et al., 2022). Based on these observations, we propose to quantify ReLU sensitivity with a _proxy density_ of \(d=0.1\).

Moreover, to avoid the compute-heavy pruning process, we leverage the idea of sensitivity evaluation before training (Lee et al., 2018). On a sampled mini batch from training data \(\mathcal{D}\), the sensitivity of the \(j^{th}\) connection with associated indication variable and vector as \(c_{j}\) and \(\mathbf{e}_{j}\), can be evaluated as,

Figure 2: Layer-wise pruning sensitivity (\(d=0.1\)) vs. normalized ReLU importance. The later layers are less sensitive to pruning, and, thus, can afford significantly more zero-valued weights as opposed to the earlier ones. On the contrary, later ReLU stages generally have higher importance.
\[\Delta\mathcal{L}_{j}(f_{\Phi}(\mathbf{\Theta};\mathcal{D})) =g_{j}(f_{\Phi}(\mathbf{\Theta};\mathcal{D}))=\frac{\partial\mathcal{L} (f_{\Phi}(\mathbf{c}\odot\mathbf{\Theta};\mathcal{D}))}{\partial c_{j}}\Big{|}_{\text{ e=1}} \tag{3}\] \[=\lim_{\delta\to 0}\frac{\mathcal{L}(f_{\Phi}(\mathbf{c}\odot\mathbf{ \Theta};\mathcal{D}))-\mathcal{L}(f_{\Phi}((\mathbf{c}-\delta\mathbf{e}_{j})\odot\mathbf{ \Theta};\mathcal{D}))}{\delta}\Big{|}_{\text{e=1}}\]

where \(\mathbf{c}\) is a vector containing all indicator variables. The \(\frac{\partial\mathcal{L}}{\partial c_{j}}\) is an infinitesimal version of \(\Delta\mathcal{L}_{j}\) measuring the impact of a change in \(c_{j}\) from \(1\to 1-\delta\). It can be computed using one forward pass for all \(j\) at once. We normalize the connection sensitivities, rank them, and identify the top d-fraction of connections. We then define the layer sensitivity \(\eta_{\mathbf{\Theta}^{i}}\) as the fraction of connections of each layer that are in the top d-fraction. For a given global ReLU budget \(r\), we then assign the \(\#\) ReLU for each layer proportional to its normalized ReLU sensitivity. The details are shown in Algorithm 1 (Fig. 3 as point 1). Note \(r^{l}_{final}\) in Algorithm 1 represents the allocated #ReLUs of layer \(l\) at the end of stage 1, with \(\mathbf{r}_{final}\) representing the set of #ReLUs for all the ReLU layers.

### ReLU Mask Identification

After layer-wise #ReLU allocation, we identify the ReLU locations in each layer's activation map. In particular, for a non-linear layer \(l\), we assign a mask tensor \(M^{l}\in\{0,1\}^{h^{l}\times w^{l}\times c^{l}}\), where \(h^{l},w^{l}\), and \(c^{l}\) represents the height, width, and the number of channels in the activation map. For a layer \(l\), we initialize \(M\) with \(r^{l}_{final}\) assigned 1's with random locations. Then we perform a distillation-based training of the PR model performing ReLU ops only at the locations of the masks with 1, while distilling knowledge from an AR model of the same architecture (see Fig. 3, point 2). At the end of each epoch, for each layer \(l\), we rank the top-\(r^{l}_{final}\) locations based on the highest absolute difference between the PR and AR model's post-ReLU activation output (averaged over all the mini-batches) for that layer, and update the \(M^{l}\) with 1's at these locations. This, on average, de-emphasizes the locations where the post-ReLU activations in both the PR and AR models are positive. We terminate mask evaluation once the ReLU mask1 evaluation reaches the maximum mask training epochs or when the normalized hamming distance between masks generated after two consecutive epochs is below a certain pre-defined \(\epsilon\) value. **Notably, there has been significant research in identifying important trainable parameters (Savarese et al., 2020; Kusupati et al., 2020; Kundu et al., 2020; 2022c;b; Babakniya et al., 2022) through various proxies including magnitude, gradient, Hessian, however, due to the absence of any trainable parameter in the ReLU layer, such methods can't be deployed in identifying important ReLU units of a layer.**
**Channel-wise ReLU mask identification.** The mask identification technique described above, creates irregular ReLU masks. To support a coarser level of granularity where the ReLU removal happens "channel-wise", we now present a simple yet effective extension of the mask identification. For a layer \(l\), we first translate the total non-zero ReLU counts to total non-zero ReLU channels as \(r_{c}^{l}=\lceil\frac{r_{r,l=1}^{l}}{h\cdot w_{l}^{l}}\rceil\). We then follow the same procedure as irregular mask identification, however, only keep top-\(r_{c}^{l}\) channels as non-zero.

### Maximizing Activation Similarity via Distillation

Once the mask for each layer is frozen, we start our final training phase in which we maximize the similarity between activation functions of our PR and AR models, see Fig. 3, point 1. In particular, we initialize a PR model with the weights and mask of best PR model of stage 2 and allow only the parameters to train. We train the PR model with distillation via KL-divergence loss (Hinton et al., 2015; Kundu and Sundaresan, 2021) from a pre-trained AR along with a CE-loss. Moreover, we introduce an AR-PR post-ReLU activation mismatch (PRAM) penalty into the loss function. This loss drives the PR model to have activation maps that are similar to that of the AR model.

More formally, let \(\Psi_{pr}^{m}\) and \(\Psi_{ar}^{m}\) represent the \(m^{th}\) pair of vectorized post-ReLU activation maps of same layer for \(\Phi_{pr}\) and \(\Phi_{ar}\), respectively. Our loss function for the fine-tuning phase is given as

\[\mathcal{L}=(1-\lambda)\underbrace{\mathcal{L}_{pr}(y,y^{pr})}_{\text{CE loss}}+\lambda\underbrace{\mathcal{L}_{KL}\left(\sigma\left(\frac{z^{ar}}{ \rho}\right),\sigma\left(\frac{z^{pr}}{\rho}\right)\right)}_{\text{KL-div. loss}}+ \frac{\beta}{2}\sum_{m\in I}\underbrace{\left\|\frac{\Psi_{pr}^{m}}{\|\Psi_{ pr}^{m}\|_{2}}-\frac{\Psi_{ar}^{m}}{\|\Psi_{ar}^{m}\|_{2}}\right\|_{2}}_{\text{PRAM loss}} \tag{4}\]

where \(\sigma\) represents the softmax function with \(\rho\) being its temperature. \(\lambda\) balances the importance between the CE and KL divergence loss components, and \(\beta\) is the weight for the PRAM loss. Similar to (Zagoruyko and Komodakis, 2016), we use the \(l_{2}\)-norm of the normalized activation maps to compute this loss.

### SENet++: Support for Ordered Channel Dropping

To yield further compute-communication benefits, we now present an extension of SENet, namely SENet++, that can perform the ReLU reduction while also supporting inference with reduced model sizes. In particular, we leverage the idea of ordered dropout (OD) (Horvath et al., 2021) to simultaneously train multiple sub-models with different fractions of channels. The OD method is parameterized by a candidate dropout set \(\mathcal{D}_{r}\) with dropout rate values \(d_{r}\in(0,1]\). At a selected \(d_{r}\) for any layer \(l\), the model uses a \(d_{r}\)-sub-model with only the channels with indices \(\{0,1,...,[d_{r}\cdot C_{l}]-1\}\) active, effectively pruning the remaining \(\{[d_{r}\cdot C_{l}],...,C_{l}-1\}\) channels. Hence, during training, the selection of a \(d_{r}\)-sub-model with \(d_{r}<1.0\in\mathcal{D}_{r}\), is a form of channel pruning, while \(d_{r}=1.0\) trains the full model. For each mini-batch of data, we perform a forward pass once for each value of \(d_{r}\) in \(\mathcal{D}_{r}\), accumulating the loss. We then perform a backward pass in which the model parameters are updated based on the gradients computed on the accumulated loss. We first train an AR model with a dropout set \(\mathcal{D}_{r}\). For the ReLU budget evaluation, we consider only the model with \(d_{r}=1.0\), and finalize the mask by following the methods in Sections 4.1 and 4.2. During the maximizing of activation similarity stage, we fine-tune the PR model supporting the same set \(\mathcal{D}_{r}\) as that of the AR model. In particular, the loss function for the fine-tuning is the same as 4, for \(d_{r}=1.0\). For \(d_{r}<1.0\), we exclude the PRAM loss because we empirically observed that adding the PRAM loss

Figure 3: Different stages of the proposed training methodology for efficient private inference with dynamic channel reduction. For example, the model here supports two channel SFs, \(S_{1}\) and \(S_{2}\). Note, similar to (Horvath et al., 2021), for each SF support we use a separate batch-normalization (BN) layer to maintain separate statistics.
for each sub-model on average does not improve accuracy. During inference, SENet++ models can be dynamically switched to support reduced channel widths, reducing the number of both ReLUs and MACs compared to the baseline model.

## 5 Experiments

### Experimental Setup

**Models and Datasets.** To evaluate the efficacy of the SENet yielded models, we performed extensive experiments on three popular datasets, CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), Tiny-ImageNet (Hansen, 2015), and ImageNet2 with three different model variants, namely ResNet (ResNet18, ResNet34) (He et al., 2016), wide residual network (WRN22-8) (Zagoruyko and Komodakis, 2016), and VGG (VGG16) (Simonyan and Zisserman, 2014). We used PyTorch API to define and train our models on an Nvidia RTX 2080 Ti GPU.

Footnote 2: On ImageNet, for comprehensive training with limited resources, we sample 100 classes from the ImageNet dataset with 500 and 50 training and test examples per class, respectively.

**Training Hyperparameters.** We performed standard data augmentation (horizontal flip and random cropping with reflective padding) and the SGD optimizer for all training. We trained the baseline all-ReLU model for 240, 120, and 60 epochs for CIFAR, Tiny-ImageNet, and ImageNet respectively, with a starting learning rate (LR) of 0.05 that decays by a factor of 0.1 at the 62.5%, 75%, and 87.5% training epochs completion points. For all the training we used an weight decay coefficient of \(5\times 10^{-4}\). For a target ReLU budget, we performed the mask evaluation for 150, 100, and 30 epochs, respectively, for the three dataset types with the \(\epsilon\) set to \(0.05\), meaning the training prematurely terminates when less than \(5\%\) of the total #ReLU masks change their positions. Finally, we performed the post-ReLU activation similarity improvement for 180, 120, and 50 epochs, for CIFAR, Tiny-ImageNet, and ImageNet respectively. Also, unless stated otherwise, we use \(\lambda=0.9\), and \(\beta=1000\) for the loss described in Eq. 4. Further details of our training hyper-parameter choices are provided in the Appendix. In Table 5, we report the accuracy averaged over three runs.

### SENet Results

As shown in Table 5, SENet yields models that have higher accuracy than existing alternatives by a significant margin while often requiring fewer ReLUs. For example, at a small ReLU budget of \(\leq 100\)k, our models yield up to \(4.15\%\) and \(7.8\%\) higher accuracy, on CIFAR-10 and CIFAR-100, respectively. At a ReLU budget of \(\leq 500\)k, our improvement is up to \(0.50\%\) and \(2.38\%\), respectively, on the two datasets. We further evaluate the communication saving due to the nonlinearity reduction by taking the per ReLU communication cost mentioned in Table 2. In particular, the communication saving reported in the \(8^{th}\) column of Table 5 is computed as the ratio of communication costs associated with an AR model to that of the corresponding PR model with reduced ReLUs. We did not report any saving for the custom models, as they do not have any corresponding AR baseline model. On Tiny-ImageNet, SENet models can provide up to \(0.3\%\) higher performance while requiring \(3.08\times\) fewer ReLUs (Table 3). More importantly, even for a high resolution dataset like ImageNet, SENet models can yield close to the baseline performance, depicting the efficacy of our proposed training method.

\begin{table}
\begin{tabular}{c|c|c|c|c|c} \hline \hline
**Model** & **Baseline** & **ReLU** & **Method** & **Test** & **Accity** & **Comm.** \\  & Acc.\(\epsilon\) & **(\%)** & **Acc.\(\epsilon\)** & **Max.\(\epsilon\)** & **Max.\(\epsilon\)** \\ \hline \hline \multirow{3}{*}{Weeks} & \multirow{3}{*}{\(66.1\)} & 142 & \multirow{3}{*}{\begin{tabular}{} \end{tabular} } & 55.9 & 0.414 & \(\uparrow\)15.7 \\  & & 208 & & **54.04** & \(0.218\) & \(7.5\times\) \\  & & 937 & & **59.02** & 0.46 & \(0.175\) & \(7.7\times\) \\  & & 917 & & & 64.66 & \(0.071\) & \(\uparrow\)2.4 \\ \cline{2-6}  & & \multicolumn{3}{c|}{} & \multicolumn{3}{c|}{} & \multicolumn{3}{c|}{} \\ \cline{2-6}  & & \multicolumn{3}{c|}{} & \multicolumn{3}{c|}{} & \multicolumn{3}{c|}{} & \multicolumn{3}{c|}{} \\ \hline \multirow{3}{*}{Weeks} & \multirow{3}{*}{\(71.94\)} & 000 & \multirow{3}{*}{\begin{tabular}{} \end{tabular} } & \multirow{3}{*}{
\begin{tabular}{} \end{tabular} } & 70.28 & 0.117 & \(\downarrow\)3.86\(\times\) \\  & & 950 & & **71.10** & \(0.075\) & \(2.43\times\) \\ \hline \end{tabular}
\end{table}
Table 4: Results with ReLU reduction at the granularity of activation channel evaluated on CIFAR-100.

\begin{table}
\begin{tabular}{c|l|c|c} \hline \hline
**Model** & **Baseline** & **ReLU** & **Method** & **Test** & **Accity** & **Comm.** \\  & Acc.\(\epsilon\) & **(\%)** & **Max.\(\epsilon\)** & **Max.\(\epsilon\)** \\ \hline \hline \multirow{3}{*}{Weeks} & \multirow{3}{*}{\(66.1\)} & 142 & \multirow{3}{*}{\begin{tabular}{} \end{tabular} } & 55.9 & 0.414 & \(\uparrow\)15.7 \\  & & 208 & & **54.04** & \(0.218\) & \(7.5\times\) \\  & & 937 & & **59.02** & \(0.26\times\) & \(0.175\) & \(7.7\times\) \\  & & 917 & & & 64.66 & \(0.071\) & \(\uparrow\)2.4 \\ \hline \multirow{3}{*}{Weeks} & \multirow{3}{*}{\(71.94\)} & \multirow{3}{*}{\(600\)} & \multirow{3}{*}{
\begin{tabular}{} \end{tabular} } & 70.28 & 0.117 & \(\downarrow\)3.86\(\times\) \\  & & 950 & & **71.10** & \(0.075\) & \(2.43\times\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results on Tiny-ImageNet and ImageNet.
**Results with activation channel level ReLU reduction.** As shown in Table 4, while trimming ReLUs at a higher granularity of activation channel level, SENet models suffer a little more drop in accuracy compared to that at pixel level. For example, at a ReLU budget of 240k, channel-level ReLU removal yields an accuracy of \(79.3\%\) compared to \(79.81\%\) of pixel-level. However, compared to existing alternatives, SENet can achieve improved performance of up to \(1.85\%\) for similar ReLUs.

### SENet++ Results

For SENet++, we performed experiments with \(\mathcal{D}_{r}=[0.5,1.0]\), meaning each training loop can yield models with two different channel dropout rates. The \(0.5\)-sub-model enjoys a \(\sim\)\(4\times\) MACs reduction compared to the full model. Moreover, as shown in Fig. 4, the \(0.5\)-sub-model also requires significantly less \(\#\)ReLUs due to reduced model size. In particular, the smaller models have \(\#\)ReLUs reduced by a factor of \(2.05\times\), \(2.08\times\), and \(1.88\times\) on CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively, compared to the PR full models, averaged over four experiments with different ReLU budgets for each dataset. _Lastly, the similar performance of the SENet and SENet++ models at \(d_{r}=1.0\) with similar ReLU budgets, clearly depicts the ability of SENet++ to yield multiple sub-models without sacrificing any accuracy for the full model._

### Analysis of Linear and ReLU Inference Latency

Table 2 shows the GC-based online ReLU operation latency is \(\sim\)\(343\times\) higher than one linear operation (multiply and accumulate), making the ReLU operation latency the dominant latency component. Inspired by this observation, we quantify the online PI latency as that of the \(N\) ReLU operations for a model with ReLU budget of \(N\). In particular, based on this evaluation, Fig. 5(a) shows the superiority of SENet++ of up to \(\sim\)\(9.6\times\) (\(\sim\)\(1.92\times\)) reduced online ReLU latency on CIFAR-100 (CIFAR-100). With negligibly less accuracy this latency improvement can be up to \(\sim\)\(21\times\). Furthermore, when \(d_{r}<1.0\), SENet++ requires fewer MACs and the linear operation latency can be significantly reduced, as demonstrated in Fig. 5(b).

### Ablation Studies

**Importance of ReLU sensitivity.** To understand the importance of layer-wise ReLU sensitivity evaluations at a given ReLU budget, we conducted experiments with evenly allocated

\begin{table}
\begin{tabular}{c|c|c|c|c|c|c|c} \hline \hline \multicolumn{2}{c|}{**min \(\leq r\leq\) max**} & **Model** & **Baseline** & **ReLU (\(\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\
ReLUs. Specifically, for ResNet18, for a ReLU budget of \(25\%\) of that of the original model, we randomly removed \(75\%\) ReLUs from each PR layer with identity elements to create the ReLU mask, and trained the PR model with this mask. We further trained two other PR ResNet18 with similar and lower # ReLU budgets with the per-layer ReLUs assigned following the proposed sensitivity. As shown in Table 6, the sensitivity-driven PR models can yield significantly improved performance of \(\sim\)\(5.76\%\) for similar ReLU budget, demonstrating the importance of proposed ReLU sensitivity.

**Choice of the hyperparameter \(\lambda\) and \(\beta\).** To determine the influence of the AR teacher's influence on the PR model's learning, we conducted the final stage distillation with \(\lambda\in[0.1,0.3,0.5,0.7,0.9]\) and \(\beta\in[100,300,500,700,1000]\). As shown in Fig. 6, the performance of the student PR model improves with the increasing influence of the teacher both in terms of high \(\lambda\) and \(\beta\) values. However, we also observe, the performance improvement tends to saturate at \(\beta\approx 1000\). Note, we keep \(\lambda=0.5\) and \(\beta=1000\), for the \(\beta\) and \(\lambda\) ablations, respectively.

## 6 Conclusions

In this paper, we introduced the notion of ReLU sensitivity for non-linear layers of a DNN model. Based on this notion, we present an automated ReLU allocation and training algorithm for models with limited ReLU budgets that targets latency and communication-efficient PI. The resulting networks can achieve similar to SOTA accuracy while significantly reducing the \(\#\) ReLUs by up to \(9.6\times\) on CIFAR-10, enabling a dramatic reduction of the latency and communication costs of PI. Extending this idea of efficient PI to vision transformer models is an interesting future research.

## 7 Acknowledgment

This work was supported in parts by Intel, DARPA under the agreement numbers HR00112190120, and FA8650-18-1-7817.

\begin{table}
\begin{tabular}{c|c|c|c|c|c|c} \hline
**Model** & **Baseline** & **ReLU** & **ReLU** & **Net** & **Acciv** & **Coann.** \\  & Accs & \(\lambda_{0}\) & Sensitivity & Accs & \(\#\) & ReLU & training \\ \hline \multirow{2}{*}{ResNet18} & \multirow{2}{*}{78.05} & 135.2 & \(\mathbf{\mathcal{F}}\) & \(70.12\) & \(0.503\) & \(\times 4\) \\  & & & & & & \\ \cline{1-1}  & & & & & & \\ \cline{1-1}  & & & & & & \\ \hline \end{tabular}
\end{table}
Table 6: Importance of ReLU sensitivity.

Figure 4: Performance of SENet++ on three datasets for various \(\#\)ReLU budgets. The points labeled A, B, C, and D corresponds to experiments of different target #ReLUs for the full model (\(d_{r}=1.0\)). For SENet++, note that a single training loop yields two points with the same label corresponding to the two different dropout rates.

Figure 5: Performance comparison of SENet++ (with \(d_{r}=1.0\) and \(0.5\)) vs. existing alternatives (a) with VGG16 and ResNet18 in terms of ReLU latency. The labels A, B, C, D correspond to experiments of different target #ReLUs for the full model (\(d_{r}=1.0\)). For SENet++, note that a single training loop yields two points with the same label corresponding to the two different dropout rates. (b) Comparison between DeepReDuce and SENet++ for a target \(\#\) ReLU budget of \(\sim\)\(50\)k with ResNet18 on CIFAR-100.

Figure 6: Ablation studies with different \(\lambda\) and \(\beta\) values for the loss term in Eq. 4.