High-Resolution Convolutional Neural Networks on Homomorphically
  Encrypted Data via Sharding Ciphertexts
Recently, Deep Convolutional Neural Networks (DCNNs) including the ResNet-20
architecture have been privately evaluated on encrypted, low-resolution data
with the Residue-Number-System Cheon-Kim-Kim-Song (RNS-CKKS) homomorphic
encryption scheme. We extend methods for evaluating DCNNs on images with larger
dimensions and many channels, beyond what can be stored in single ciphertexts.
Additionally, we simplify and improve the efficiency of the recently introduced
multiplexed image format, demonstrating that homomorphic evaluation can work
with standard, row-major matrix packing and results in encrypted inference time
speedups by $4.6-6.5\times$. We also show how existing DCNN models can be
regularized during the training process to further improve efficiency and
accuracy. These techniques are applied to homomorphically evaluate a DCNN with
high accuracy on the high-resolution ImageNet dataset, achieving $80.2\%$ top-1
accuracy. We also achieve an accuracy of homomorphically evaluated CNNs on the
CIFAR-10 dataset of $98.3\%$.
Vivian Maloney, Richard F. Obrecht, Vikram Saraph, Prathibha Rama, Kate Tallaksen
High-Resolution Convolutional Neural Networks on Homomorphically Encrypted Data via Sharding Ciphertexts

###### Abstract

Recently, Deep Convolutional Neural Networks (DCNNs) including the ResNet-20 architecture have been privately evaluated on encrypted, low-resolution data with the Residue-Number-System Cheon-Kim-Kim-Song (RNS-CKKS) homomorphic encryption scheme. We extend methods for evaluating DCNNs on images with larger dimensions and many channels, beyond what can be stored in single ciphertexts. Additionally, we simplify and improve the efficiency of the recently introduced multiplexed image format, demonstrating that homomorphic evaluation can work with standard, row-major matrix packing and results in encrypted inference time speedups by \(4.6-6.5\times\). We also show how existing DCNN models can be regularized during the training process to further improve efficiency and accuracy. These techniques are applied to homomorphically evaluate a DCNN with high accuracy on the high-resolution ImageNet dataset for the first time, achieving \(80.2\%\) top-1 accuracy. We also achieve the highest reported accuracy of homomorphically evaluated CNNs on the CIFAR-10 dataset of \(98.3\%\).

## 1 Introduction

Deep learning has emerged as a powerful tool for solving image processing tasks due to its ability to automatically learn relevant features from raw data. Convolutional Neural Networks (CNNs), which are a type of deep learning model specifically designed for image processing, have achieved state-of-the-art performance on a variety of image processing tasks such as image classification [13], object detection [17], and segmentation [20].

Fully homomorphic encryption (FHE) [9; 19] is a technique enabling computation directly on encrypted data, and in particular, enabling Privacy Preserving Machine Learning (PPML). FHE has potential societal impact in applications where user and data privacy are critical, such as in cloud computing, healthcare analytics, and defense applications. However, adoption of FHE has been limited due to the speed of existing FHE neural network inference algorithms, and limitations of FHE itself. Previous work uses narrow or shallow DCNNs on low-resolution data, often using nonstandard activation functions, since FHE can only evaluate polynomials. Furthermore, it is challenging to ensure that polynomial approximations of activation functions are suitably accurate.

Key contributions of this work are summarized as follows:

* We design and implement efficient homomorphic convolution and pooling algorithms, which have been parallelized and handle large inputs and channels via sharding techniques.
* We apply these algorithms to construct three families of ResNet architectures, achieving the highest homomophically evaluated accuracy on CIFAR-10 and ImageNet-1k while re
ducing the inference latency relative to the previous state-of-the-art. We also do not observe any degradation of encrypted model accuracy relative to its unencrypted counterpart.
* We propose a training technique to reduce the input range to our activation functions by penalizing the kurtosis of the distributions of BatchNorm outputs, allowing efficient homomorphic polynomial approximation of the GELU activation function.

## 2 Background

Homomorphic encryptionRNS-CKKS [6; 7] is an FHE scheme that supports arithmetic over encrypted vectors of fixed-point numbers. Ciphertexts in this scheme are elements in the ring \(R_{Q}^{2}\), where \(R_{Q}=\mathbb{Z}_{Q}[x]/(x^{2N}+1)\) and \(Q\) is a large integer, and \(2N\) is called the _ring dimension_. Each such ciphertext has \(N\)_slots_, each of which stores a single real number, so it is useful to conceive of a ciphertext as a vector. Ciphertext vectors support vectorized addition and multiplication operations, as well as cyclic rotations. We pack images into RNS-CKKS ciphertexts.

Each ciphertext has a _level_, or maximum number of multiplications that can be applied before decryption error becomes too high; each multiplication reduces the level by one. The ciphertext level is restored through _bootstrapping_, though this is a time-consuming operation to be used sparingly.

Threat ModelThe threat model assumed is similar to previous PPMLs [5; 15]. We encrypt the input image but not the model weights. A client homomorphically encrypts data it wishes to send, which is then sent to a server for processing. The server performs inference on the encrypted data directly, sending back the encrypted inference result to the client. Since it is assumed that only the client holds the secret key, only they can decrypt the result, which guarantees privacy from the server. Because the server does not see the decrypted inference result, the Li-Macciato attack [16] is not applicable and we do not need to take noise flooding into account in our parameter selection.

## 3 Related Work

Early work on encrypted machine learning evaluated narrow and shallow CNNs with nonstandard activation functions on low-resolution data [5; 10]. Recent papers have begun evaluating larger CNNs with standard design features on encrypted data. Prior work on PPML most similar to ours are Multiplexed Parallel Convolutions [15] and TileTensors [1]. Multiplexed Parallel Convolutions homomorphically evaluates deep but narrow CNNs with standard activation functions on low-resolution data. TileTensors homomorphically evaluates shallow CNNs with nonstandard activation functions on high-resolution data. In this work, we homomorphically evaluate wide and deep CNNs with standard activation functions on high-resolution data.

TileTensors uses concepts similar to our sharding approach to perform inference on \(224\times 224\) images using a modified AlexNet. They rely on shallow CNNs and do not perform the bootstrapping necessary to incorporate standard activation functions, instead relying on the same nonstandard activation function used in CryptoNets [10] and LoLa Nets [5], which is unsuited for DCNNs.

We improve on Multiplexed Parallel Convolutions, hereby defined as the multiplexed ResNet family, by supporting high-resolution images and wide channels that do not fit into a single ciphertext, as well as simplified packing. We also introduce a novel training regularization technique, enabling more efficient homomorphic evaluation of non-linear activations. Our implementation performs encrypted inference on a multiplexed ResNet-20 architecture \(4.6\times\) faster than Ref. [15]. We homomorphically evaluate wide ResNet architectures not supported by the multiplexed algorithms, and achieve significantly higher accuracy than multiplexed architectures on standard datasets.

## 4 Homomorphic Neural Network Operators

Algorithms have been carefully designed to minimize the number of encrypted multiplication and rotation operations to minimize latency. An _image_ consists of many _channels_. All dimensions are assumed to be powers of two, and each channel is assumed to be square in shape. The approach is adaptable to dimensions not powers of two with appropriate rescaling or zero padding. Given an image with \(c\) channels of size \(m\times m\), we homomorphically encrypt and represent it with RNS-CKKS
vectors. To encrypt an image into a ciphertext vector of size \(m^{2}c\), each channel \(M^{i}\) is represented in row-major order, and they are concatenated to obtain a single plaintext vector.

Sharding and Encrypting an ImageIn RNS-CKKS, storage capacity of a single ciphertext is determined by the ring dimension of the scheme, and is typically in the range \(2^{14}\) to \(2^{16}\). If a \(c\times m\times m\) tensor does not fit into a single ciphertext, channels are spread across _multiple_ ciphertexts, such that each ciphertext stores a subset of channels. Here, each ciphertext vector is called a _shard_, and the maximum amount of data storable in a shard is called the _shard size_. The performance of the scheme degrades with increasing ring dimension, so increasing the ring dimension to avoid sharding would negatively impact the efficiency of encrypted inference.

We distinguish the two cases of _image shards_ and _channel shards_. For _image shards_, a shard is large enough to hold at least one channel (\(m^{2}\leq s\)), but multiple shards are needed to store all channels (\(m^{2}c>s\)). See Figure 0(a) for an example of image shards. For _channel shards_ each channel must be split up across multiple shards (\(m^{2}>s\)), so that each shard contains a set of consecutive rows from a single channel. See Figure 0(b).

Duplicating and Permuting ChannelsIf an image does not fill a shard, its channels are _duplicated_. When \(s>m^{2}c\), we define a _duplication factor_ given by \(d=s/m^{2}c\), and place \(d\) copies of each channel when concatenating them together. \(d\) is tracked with the encrypted image as metadata. Our implementation of average pooling can _permute_ input channels. If one tracks the channels' order with a permutation defining the correct order, subsequent convolution operations can also be computed correctly. Therefore, we attach a channel permutation as metadata to an encrypted image.

### Convolution

We describe how to homomorphically convolve a single matrix with a single kernel, using same padding and a stride of \(1\); this does not change the channel's dimensions. Convolution is typically thought of as sliding a kernel over as matrix. However, one may also think of convolution as fixing the kernel, and sliding the matrix, which is a more useful visual in what follows. We formalize this observation and use it to compute convolutions. Denote \(\mathcal{S}_{k,\ell}\) on matrix \(M\) as a function that shifts rows up by \(k\) and columns left by \(\ell\). \(\mathcal{S}_{k,\ell}\) adds zeros when elements are shifted off the matrix. Then:

\[M*K=\sum_{k=-\kappa/2}^{\kappa/2}\sum_{\ell=-\kappa/2}^{\kappa/2}K_{k,\ell} \cdot\mathcal{S}_{k,\ell}(M). \tag{1}\]

See Figure 1. \(\mathcal{S}_{k,\ell}\) is implemented homomorphically: shifting a row-major matrix by one column is done by rotating the ciphertext vector by \(1\), while shifting by a row is done by rotating by \(m\). Wrap-around elements are zeroed out by multiplying the ciphertext vector with an appropriate binary mask. This allows us to homomorphically compute \(\mathcal{S}_{k,\ell}(M)\) for any shifts \(k\) and \(\ell\). To multiply \(\mathcal{S}_{k,\ell}(M)\) by the scalar \(K_{k,\ell}\), we create a vector of size \(m^{2}\) and multiply \(\mathcal{S}_{k,\ell}(M)\) elementwise with this vector. In practice, the multiplications for shift masking and those for kernel element multiplication are combined non-homomorphically before being applied homomorphically.

With a Single ShardRecall that to convolve a \(c\)-channel image with a single filter, \(c\) matrix convolutions are individually computed, and the results are summed. An image is typically convolved with multiple filters to produce multiple channels. Convolutions are computed in parallel all at once.

Figure 1: Illustrations of image sharding and channel sharding.
Given an image \(M\), denote \(M^{f}_{ij}\) as the \((i,j)\)-th element in the \(f\)-th channel of \(M\). Filters \(K\) ordinarily have dimensions \(c_{i}\times c_{o}\times m\times m\), so that \(K^{fg}_{ij}\) is \((i,j)\)-th element in the kernel convolved with the \(f\)-th input channel used to compute the \(g\)-th output channel. We begin with a \(1\times 1\) kernel size, in which case \(K^{fg}\) is the single-element kernel applied to the \(f\)-th input channel, to compute the \(g\)-th output channel. We further assume that \(M\) fits in exactly one shard, and that \(c_{i}=c_{o}=c\), so that \(M*K\) also occupies one shard. Then the \(g\)-th channel of \(M*K\) is given by Equation 2:

\[(M*K)^{g}=\sum_{r=0}^{c-1}K^{r+g,g}\cdot M^{r+g} \tag{2}\] \[\bigparallel_{g=0}^{c-1}K^{r+g,g}\cdot M^{r+g} \tag{3}\]

where index arithmetic above is modulo \(c\). We compute all \(c\) output channels simultaneously. Given \(0\leq r<c\), the \(r\)-th _partial convolution_ is defined in Equation 3. The full convolution is obtained by summing over partial convolutions:

\[M*K=\sum_{r=0}^{c-1}\bigparallel_{g=0}^{c-1}K^{r+g,g}\cdot M^{r+g}=\sum_{r=0} ^{c-1}\left(\bigparallel_{g=0}^{c-1}K^{r+g,g}\cdot\bigparallel_{g=0}^{c-1}M^ {r+g}\right). \tag{4}\]

See Figure 1 for a simple illustration of summing partial convolutions. Each summand corresponds to a single rotation of the ciphertext \(M\) by \(r\cdot m\cdot m\) positions.

When working with larger kernels, the prior approaches combine to compute the \(g\)-th output channel:

\[(M*K)^{g}=\sum_{r=0}^{c-1}\sum_{k=-\kappa/2}^{\kappa/2}\sum_{\ell=-\kappa/2}^{ \kappa/2}K^{r+g,g}_{k,\ell}\cdot\mathcal{S}_{k,\ell}(M^{r+g}). \tag{5}\]

Rotations \(\mathcal{S}_{k,\ell}(M^{r+g})\) are computed once and cached. As with \(1\times 1\) kernels, we use partial convolutions to compute all \(c\) channels at once.

Rather than directly implement strided convolution as in Ref. [15], we instead compose an unstrided convolution with downsampling described in Section 4.2. This preserves the row-major order format and avoids multiplexed packing, and increases efficiency, as the multiplexed convolution algorithm of Ref. [15] has a multiplicative depth of 2, while we only use a single multiplicative level.

Figure 2: (a) Partial convolution computation for a \(4\)-channel image convolved with a \(1\times 1\) kernel. (b) A single convolution computed by shifting the matrix. (c) Shifting rows from channel shards into adjacent ones.
With Image ShardsLet \(M\) be an image of dimension \(c_{i}\times m\times m\), split across \(t\) shards, denoted as \([M]_{0},\dots,[M]_{t-1}\), implying a shard size \(s=\frac{m^{2}c_{i}}{t}\). Suppose we want to convolve \(M\) with filters \(K\) with dimensions \(c_{i}\times c_{o}\times m\times m\). Then the \(v\)-th output shard, \([M*K]_{v}\), is computed as:

\[[M*K]_{v}=\sum_{u=0}^{t-1}[M]_{u}*K^{\iota(u),\iota(v)}, \tag{6}\]

where \(\iota(u)\) is the index interval \(\iota(u)=[z\cdot u:z\cdot(u+1)]\), and \(z=s/m^{2}\), or the number of channels per shard. Intuitively, each single convolution in the summand above is computed using the approach in the previous section 4.1, slicing \(K\) accordingly, and summing up the results. With a shard size of \(s\), \(M*K\) is packed into \(c_{0}m^{2}/s\) shards, and \(v\) ranges over this.

Single Shard with Duplication and PermutationConvolution must work with a shard with \(d\)-duplicated channels. Filters \(K\) can be duplicated accordingly, but we instead index into \(d\) times when computing \(M*K\). Channels can also be permuted by pooling (see 4.2). In this case, the image passed from the previous layer is also assumed to return a permutation \(\tau\) defining the correct channel order. To compute a convolution using this permutation, any time we were to index into the filter \(K\) at input channel \(i\) (so \(K^{i}\)), we instead index into \(K\) at \(\tau(i)\) (so \(K^{\tau(i)}\)).

With Channel ShardsConvolving a channel-sharded image results in a channel-sharded image. Output channels are computed independently from one another, so we initially focus on convolving a shard of a single channel with a single kernel. Let \(M^{f}\) be the \(f\)-th input channel of image \(M\), which we convolve with a single kernel \(K\). Let \([M^{f}]_{u}\) be the \(u\)-th shard. We cache all _cyclic_ rotations \(\mathcal{S}_{k,\ell}([M^{f}]_{u})\), for \(k,\ell\) ranging over the indices of \(K\). \([M^{f}*K]_{v}\) is computed from the cached rotations of the input shards.

Shifting channels requires shifting all associated shards simultaneously. Shifting columns is accomplished by shifting each shard independently. When shifting rows, one needs to shift rows of one shard into an adjacent shard. Each row shift is constructed from two cached rotations (with the exception of first and last shards). See Figure 1(a) showing how rows are shifted between shards.

Each output channel is computed by summing over row and column shifts, and each summand is itself a sum of two kernel-masked shards. That is:

\[[M^{f}*K]_{v}=\sum_{k=-\kappa/2}^{\kappa/2}\sum_{\ell=-\kappa/2}^{\kappa/2} \mathfrak{m}_{k,\ell}(K_{k,\ell})\cdot\mathcal{S}_{k,\ell}([M^{f}]_{v})+ \overline{\mathfrak{m}_{k,\ell}}(K_{k,\ell})\cdot\mathcal{S}_{k,\ell}([M^{f}] _{v+\operatorname{sign}k}) \tag{7}\]

where \(\mathfrak{m}_{k,\ell}(x)\) is the vector given by shard-size-many elements of all \(x\), multiplied by the binary mask used in the shift operator \(\mathcal{S}_{k,\ell}\), and \(\overline{\mathfrak{m}_{k,\ell}}(x)\) is its complement. Then, to compute one shard \([M*K]_{v}\) of a single channel, we simply sum the shards \([M^{f}*K]_{v}\) over the input channels \(f\). Each such shard is computed independently done in parallel, concluding channel-sharded convolution.

### Average Pooling

We implement an average pooling operation with a \(2\times 2\) window; this increases the channel capacity of each shard by a factor of four. Our implementation preserves the format described previously, avoiding multiplexed packing used in Ref. [15], which does not rearrange pixels after downsampling.

With Image ShardsThere are up to three steps involved with pooling: _downsample_, which computes the average pool but leaves the original number of shards intact; _consolidate_, which reduces the number of shards; and _duplicate_, which duplicates channels if there is a single shard remaining.

In the _downsampling_ step, we convolve each channel with a \(2\times 2\) kernel of \(1\)s (as we would with homomorphic convolutions). This replaces each \(2\times 2\) window in each channel with the sum of the elements in the window. Next, we want to select only one of four elements in the new \(2\times 2\) windows; we choose the top-left element. The following is how we operate on individual channels \(M\), but generalizes to applying the operations to all channels within each shard simultaneously.

We _horizontally reduce_ the elements in channels of each shard, which is done with masking and summing over the channels of each shard, as in Equation 8:
\[M^{\prime}=\sum_{i=0}^{(m-1)/2}(M\cdot\mathfrak{m}_{i})\ll i\qquad\text{(8)}\qquad M ^{\prime\prime}=\sum_{j=0}^{(m-1)/2}(M^{\prime}\cdot\mathfrak{m}_{j})\ll 3i \cdot m/2 \tag{9}\]

where \(\mathfrak{m}_{i}\) is the binary mask that selects elements in the \(i\)-th column of each channel \(M\), and \(\ll\) (\(\gg\)) denotes ciphertext rotation to the left (right) by \(i\) slots. Then, we _vertically reduce_ each \(M^{\prime}\), as in Equation 9, where \(\mathfrak{m}_{j}\) is the binary mask that selects the left half of \(2j\)-th row in \(M^{\prime}\). See Figures 3.

After downsampling, each \(m\times m\) channel of the resulting shards contains only \(m/2\times m/2\) non-zero elements, all packed on the left-hand side. If we started with four or more shards, then we _consolidate_ the remaining non-zero elements into a quarter as many shards. This is done by rotating the shards from the previous step, and summing each group of four consecutive shards.

\[S=S_{0}+(S_{1}\gg m^{2})+(S_{2}\gg 2m^{2})+(S_{2}\gg 3m^{2}). \tag{10}\]

Starting with two image shards, we only have two summands in the above, and with one shard, there is no consolidation step. Consolidating shards results in channels out-of-order. See Figure 2(c). If we downsampled from two shards or fewer, then the resulting non-zero elements in the (single) consolidated shard would not fill up the entire shard, so we _duplicate_ the shard's channels:

\[S^{\prime}=S+(S\gg m^{2}/4)+(S\gg 2m^{2}/4)+(S\gg 3m^{2}/4). \tag{11}\]

With two shards, we duplicate by a factor of two, so the above would only have two summands.

With Channel ShardsChannel shards are downsampled individually, and every set of four consecutive shards is consolidated into one. In the edge case where the input image has one channel with two shards, we need to duplicate the resulting single shard by a factor of two. Pooling a channel-sharded image never results in an output with permuted channels.

### Other Layers

Batch NormalizationAt inference time, batch normalization is an affine transformation, which is expressible as additions and multiplications, so can be implemented homomorphically. These are folded into kernel element multiplication and bias addition in the previous convolution, respectively.

LinearEvaluation of a linear layer is a matrix multiplication of the previous layer's output with the weights of the linear layer. Each element of a matrix multiplication is computed as a dot product. The dot product of one vector with another is computed by first taking their elementwise product, then summing the elements of the resulting vector. Elements of vector \(v\) are summed by rotating over its slots, and adding the rotated vector to the original one. The result is a vector whose elements are all \(\Sigma_{i}v_{i}\), and is done in logarithmically many rotations. We get a single activation in the linear

Figure 3: Steps involved in a pooling operation. Duplication is not depicted.
layer's output, and repeat for each activation in the output of the linear layer. Activations are then masked and summed into a single ciphertext.

ResNets often pool each \(m\times m\) input channel to a single pixel, and apply a linear layer at the end. In general, the pool could use a window size larger than \(2\times 2\), which we have not implemented directly. We fuse pool and linear into a _pool-linear_. The linear layer's weights are duplicated as though it were operating on channels of size \(m\times m\), and we divide by a normalization factor of \(m^{2}\).

Gaussian Error Linear Unit (GELU)Non-linear activation functions are computed in RNS-CKKS through polynomial approximation. The polynomial degree and hence latency increases when the approximation must be accurate over a wide range. We introduce novel terms to the loss function during training to encourage hidden layer outputs to match the mean, variance, and kurtosis statistical moments of a Gaussian distribution, constraining the range over which the activation needs to be accurately computed. This allows more efficient low-degree polynomial approximation while minimally impacting model accuracy.

We use a GELU activation function since it is more amenable to polynomial approximations for fast homomorphic evaluation. We homomorphically compute a 59-degree polynomial approximation of GELU in a numerically stable way with a shallow arithmetic circuit by expanding the polynomial in a Chebyshev basis. Details on polynomial approximation of GELU and kurtosis regularization can be found in the Appendix.

## 5 Empirical Results

We use OpenFHE's implementation [3] of FHE with RNS-CKKS to implement the neural network operators described in Section 4 in C++, which are then thinly wrapped with Python bindings to build neural network architectures. Weights are loaded using PyTorch's API, though the approach is indepedent of deep learning framework. OpenMP is used to leverage parallelism from multicore CPUs. As our main focus is on fast encrypted inference of trained models rather than the unencrypted training process, we defer most of the details on the training configuration to the Appendix.

Experiments for ResNet-9 and multiplexed ResNets were run on a machine with a hyperthreaded AMD Ryzen Threadripper 3970X 32-core processor, 128 GB of memory, and an Ubuntu 22.04.2 operating system. Experiments for the encrypted ResNet-50 were run on a server with an AMD EPYC 7742 64-core processor, 800 GB of memory, and RHEL 7.9.

### Datasets

We perform image classification on CIFAR-10, CIFAR-100, and ImageNet, using various ResNets to evaluate the performance of our homomorphic neural network operators. CIFAR-10 and -100 contain \(32\times 32\) color images in 10 and 100 classes, respectively [12]. ImageNet-1k is a much larger scale dataset containing over 1.2 million high-resolution images with 1000 different classes [21], and is typically resized to \(224\times 224\) during inference, though this does not match our assumption that dimensions are powers of two. We evaluate two different models on ImageNet-1k resized to resolutions of both \(128\times 128\) and \(256\times 256\).

### Architectures

We modify DCNN architectures to decrease encrypted inference latency without adversely affecting model accuracy. We use \(2\times 2\) average pooling with stride \((2,2)\), and the GELU activation function. We train models with kurtosis regularization as described in the previous section, and more extensively in the Appendix.1.

Footnote 1: If using kurtosis-regularized GELU is not an option, such as when evaluating pre-existing models, our algorithms are compatible with any approach for computing ReLU over a wider range, such as higher-degree polynomial approximation or the approach in Ref. [14].

We homomorphically evaluate three classes of ResNets on CIFAR-10 and -100. We first evaluate the narrow deep multiplexed ResNet family used in the previous state-of-the-art for homomorphic DCNNs Ref. [15], as well as a wide ResNet-9 architecture taken from DAWNBench [8], and finally a fine-tuned version of the wide and deep ImageNet-1k ResNet-50 v1.5 [11]. The wide ResNet-9
and -50 achieve substantially higher accuracy than the multiplexed family, achieving a best accuracy of 94.7% and 98.3% on CIFAR-10, respectively, surpassing the 92.95% reported in Ref. [15] for a multiplexed ResNet-110 and the 92.8% we achieved for a multiplexed ResNet-56.

Our ImageNet-1k architecture is modified ResNet-50 v1.5 [11] with GELU and average pooling. This is a wide architecture, using between 64 and 2048 channels. On ImageNet-1k, we train and evaluate ResNet-50 on images resized to \(128\times 128\) and \(256\times 256\), respectively. The \(256\times 256\) model requires both channel shards and image shards, while the \(128\times 128\) model only requires image shards. As such, this illustrates a trade-off between model accuracy and inference time for image resolution. The resolution during training was set according to the FixRes [22] optimization, where the training resolution is \(3/4\) of evaluation resolution to account for data augmentation.

### Encrypted Inference Discussion

For the encrypted ResNet-50, we used a RNS-CKKS ring dimension of \(2^{16}\) and shard size of \(2^{15}\) with 59-bit scaling factors and a multiplicative depth of 34. When evaluating the multiplexed ResNets and ResNet-9, we used a lower shard size of \(2^{14}\). This lower shard size trades slower initial layers for faster later layers and bootstrapping operations, and improved the encrypted latency for these narrower architectures. These parameters suffice for a standard 128-bit security level [2]. The distributions prior to GELU are analyzed in order to determine a safe bound for our polynomial approximations; see the Appendix for details. For each model, runtime experiments are collected for 25 inferences; for each run, the runtimes for each algorithm are summed, and then the average is displayed in Tables 2 and 3, where the quoted error is the standard deviation in the total runtime. ResNet-9 and -50 models, which allow the channel dimension to substantially grow, spend less relative time bootstrapping when compared to the multiplexed ResNet family.

During inference on ImageNet-1k, ResNet-50 at 128 resolution uses a maximum of 32 shards, and at 256 resolution uses a maximum of 128 shards. On CIFAR-10, ResNet-50 uses a maximum of 16 shards. Due to channel size, inference on 256 resolution requires the use of channel shards, and has a \(2.9\times\) slower latency. However, note that ResNet-50 on 256 resolution has a \(6.1\%\) higher accuracy, so in this case, using higher resolution images produces a better classifier.

The _logit residual_, which is the difference between decrypted and unencrypted logits, generally form tight Gaussian distributions centered at zero. By using GELU and a small input range, we decreased the noise from bootstrapping and the polynomial approximation. This is reflected in the increased precision of the logit residual distributions, which has standard deviations at the \(10^{-4}-10^{-2}\) level when fit to a Gaussian, see Table 1 in the Appendix for more details. We ran 1000 inferences with ResNet-20 on CIFAR-10, and all encrypted predictions match respective unencrypted predictions; this is an improvement over Ref. [15], where the encrypted classification accuracy is \(0.1-0.5\)% lower than the unencrypted accuracy. Furthermore, the difference in the top-2 logits between the encrypted and unencrypted ResNet-20 are examined, yielding Gaussian standard deviations at the

\begin{table}
\begin{tabular}{l l r r} \hline \hline Dataset & Model & Average Accuracy (\%) & Best Accuracy (\%) \\ \hline CIFAR-10 & ResNet-9 & \(94.5\pm 0.1\) & 94.7 \\  & ResNet-50 & \(98.3\) & 98.3 \\  & ResNet-20* & \(90.6\pm 0.3\) & 91.0 \\  & ResNet-32* & \(92.2\pm 0.2\) & 92.5 \\  & ResNet-44* & \(92.2\pm 0.1\) & 92.3 \\  & ResNet-56* & \(92.8\pm 0.2\) & 93.0 \\  & ResNet-110* & \(92.7\pm 0.2\) & 92.8 \\ \hline CIFAR-100 & ResNet-9 & \(74.9\pm 0.2\) & 75.3 \\  & ResNet-32* & \(66.6\pm 0.4\) & 67.0 \\ \hline ImageNet-1k & ResNet-50 @ 128 & \(74.1\) & 74.1 \\  & ResNet-50 @ 256 & \(80.2\) & 80.2 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Model accuracy is averaged over five runs for all architectures except ResNet-50, and the quoted error is the standard deviation. The (*) represents our implementation of the multiplexed architectures found in Ref. [15]. Due to long training times, ResNet-50s are only trained once.
\(10^{-4}\) level. Thus, using kurtosis and GELU allows us to perform faster and more reliable encrypted inference.

As further discussed in the Appendix, we determined that the logit error is mainly due to bootstrapping noise. By applying MetaBTS [4] to reduce bootstrapping noise we further increased logit precision by a factor of \(20\times\) at the expense of a \(1.7\times\) increase in latency.

## 6 Conclusion and Future Work

We have successfully constructed three families of ResNet architectures that may be evaluated homomorphically: 1) the multiplexed family of architectures [15], 2) the ResNet-9 bag-of-tracks architectures [8], and 3) the popular ResNet-50 architecture [11]. Models have been homomorphically evaluated on a variety of standard datasets, including CIFAR-10, CIFAR-100, and ImageNet-1k. We proposed a training time technique to regularize the range of inputs to the GELU activation function by penalizing the fourth order statistical moment of the outputs of the BatchNorm distributions; this technique allows us to efficiently approximate the GELU function with polynomials under homomorphic constraints. When runtimes are compared to the previously reported runtimes of the multiplexed family, we observe a speedup on the previous state-of-the-art by approximately \(4.6-6.5\times\) without any classification accuracy degradation. We also report the highest homomorphically encrypted accuracy on CIFAR-10 and ImageNet-1k of \(98.3\%\) and \(80.2\%\), respectively.

Future work includes extending our models to more advanced tasks, such as encrypted object detection with the YOLO [18] family of architectures and sensitive document analysis with encrypted transformers [23]. Parallelization in this work was achieved with using multicore CPUs, but vectorized addition and multiplication operations on ciphertexts vectors could be ported to GPUs (or other hardware accelerators) to further accelerate computation and minimize latency.
