# HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-
Efficient Private Inference

Tianshi Xu1, Meng Li1,2,Runsheng Wang1
1 School of Integrated Circuits, Peking University
2 Institute for Artificial Intelligence, Peking University
<EMAIL_ADDRESS>
###### Abstract

Secure two-party computation with homomorphic encryption (HE) protects data
privacy with a formal security guarantee but suffers from high communication
overhead. While previous works, e.g., Cheetah, Iron, etc, have proposed
efficient HE-based protocols for different neural network (NN) operations,
they still assume high precision, e.g., fixed point 37 bit, for the NN
operations and ignore NNs’ native robustness against quantization error. In
this paper, we propose HEQuant, which features low-precision-quantization-
aware optimization for the HE-based protocols. We observe the benefit of a
naive combination of quantization and HE quickly saturates as bit precision
goes down. Hence, to further improve communication efficiency, we propose a
series of optimizations, including an intra-coefficient packing algorithm and
a quantization-aware tiling algorithm, to simultaneously reduce the number and
precision of the transferred data. Compared with prior-art HE-based protocols,
e.g., CrypTFlow2, Cheetah, Iron, etc, HEQuant achieves $3.5\sim 23.4\times$
communication reduction and $3.0\sim 9.3\times$ latency reduction. Meanwhile,
when compared with prior-art network optimization frameworks, e.g., SENet,
SNL, etc, HEQuant also achieves $3.1\sim 3.6\times$ communication reduction.

##  1 Introduction

Figure 1: The acceleration achieved through network quantization (from 32-bit
to 4-bit) on Bit Operations (BOPS), GPU, prior-art 2PC frameworks Cheetah and
Cheetah with naive quant as well as our proposed framework.

The last decade has witnessed the rapid evolution of deep learning (DL) as
well as its increasing adoption in sensitive and private applications,
including face authentication [1], medical diagnosis [22], personal assistant
[3], etc. Privacy emerges as a major concern and leads to a growing demand for
privacy-preserving DL [7, 9, 15, 29].

Secure two-party computation (2PC) based on homomorphic encryption (HE) has
recently been proposed and attracted a lot of attention [18, 8, 24]. It helps
to solve the dilemma between the DL model owner and the data owner: while the
two parties want to jointly apply the model to the data, they do not want to
give out the model or data directly. 2PC enables the computation and permits
cryptographically-strong privacy protection [16, 21, 14, 29].

However, the privacy protection of the 2PC-based inference is achieved at the
cost of high communication complexity since numerous high-precision
ciphertexts (e.g., 109-bit ciphertext for Cheetah [18]) are transferred
between the two parties. Previous works, e.g., Cheetah [18], Iron [16], Falcon
[50], etc, have proposed efficient HE-based protocols for representative deep
neural network (DNN) operations like convolutions, matrix multiplications,
etc. However, they all assume high-precision plaintext operations (e.g.,
37-bit plaintext for Cheetah [18]) are required and ignore the native
robustness of deep neural networks (DNNs) towards quantization error [13]. As
shown in Figure 1, prior works have shown that 4-bit quantization can achieve
plausible accuracy with significant compute reduction [36, 51, 2, 52, 10],
existing protocols like Cheetah cannot leverage such property for
communication reduction.

While a naive combination of quantization and HE can help reduce the
communication as shown in Figure 1, we find further efficiency improvement is
challenging: on one hand, to ensure correct decryption, the plaintext modulus
cannot be reduced arbitrarily, leading to saturated efficiency gain for low-
precision quantized networks, e.g., 4-bit or below; on the other hand, the
input and output communication for a convolution layer is highly imbalanced
while how to further reduce the output communication remains unclear.

In this paper, we propose a HE-based 2PC framework that is optimized for low-
precision quantized DNN inference, named HEQuant. HEQuant features two main
techniques to fully leverage the DNN quantization. First, HEQuant proposes an
intra-coefficient packing algorithm to pack multiple input elements into each
coefficient and allows further communication reduction given the saturated
plaintext modulus. Moreover, HEQuant further proposes a novel quantization-
aware tiling strategy for linear operations that significantly reduces output
communication. Our main contributions can be summarized as follows:

  * •

We observe existing HE-based 2PC frameworks are not optimized for quantization
and propose HEQuant, the first 2PC framework that marries quantization and HE
for efficient private inference.

  * •

We propose an intra-coefficient packing algorithm and a quantization-aware
tiling algorithm to further improve the communication efficiency for low-
precision DNNs.

  * •

HEQuant outperforms state-of-the-art (SOTA) HE-based 2PC frameworks, including
CrypTFlow2, Cheetah and Falcon with $3.5\sim 23.4\times$ communication
reduction and $3.0\sim 9.3\times$ latency reduction on CIFAR-100, Tinyimagenet
and ImageNet datasets. Compared with prior-art network optimization
algorithms, including ReLU-optimized networks and network pruning methods,
HEQuant achieves $3.5\times$ communication reduction and $4.3\times$ latency
reduction on average with higher accuracy.

##  2 Preliminaries

Table 1: Notations used in the paper.

Notations | Meanings
---|---
$\lceil\cdot\rceil$,$\lfloor\cdot\rfloor$,$\lfloor\cdot\rceil$ | Ceiling, flooring, and rounding operations
$\mathrm{Enc}(\cdot),\mathrm{Dec}(\cdot)$ | Homomorphic encryption and decryption
$\boxplus,\boxminus,\boxtimes$ | Homomorphic addition, subtraction and multiplication
$P,Q$ | Plaintext modulus, and ciphertext modulus
$N,p,q$ | Polynomial degree, bit width of plaintext
($p=\lceil\log_{2}P\rceil$) and ciphertext ($q=\lceil\log_{2}Q\rceil$)
$b_{x},b_{w},b_{acc}$ | Bit width of activation, weight, and accumulation
$H,W,C$ | Height, width, and channels of Conv2D input tensor
$R,K$ | Kernel size and the number of kernels of Conv2D filters

###  2.1 Threat Model

We focus on private DNN inference involving two parties, i.e., a server and a
client. The server holds the model with private weights, while the client
holds private inputs [18, 16, 50]. The model architecture, including the
number of layers as well as the types, dimensions, and bit widths for each
layer, is publicly known to both parties [38, 45]. At the end of the protocol
execution, the client acquires the inference results without leaking any
information to the server. Consistent with previous works [14, 19, 46, 39], we
adopt an honest-but-curious security model where both parties adhere to the
protocol’s specifications but also attempt to learn more than permitted.
Hence, malicious attacks like adversarial attacks [33] are not the focus of
the paper.

###  2.2 HE-based 2PC Inference

Table 2: Comparison with previous works. The Comm. is measured by Conv. with
dimension (H,W,C,K,R)=(14,14,256,256,3) and ReLU with input dimension
(14,14,256).

Framework | Protocol Opt. | Network Opt. | Conv. Comm. | ReLU Comm.
---|---|---|---|---
[6, 20, 30, 5] | - | ReLU Reduction | - |  $0.6\sim 1$ MB
[35] | - | Network Pruning | 9 MB | 2 MB
SiRNN[46] |  Quantized MatMul Bit Extension/Trunc |  16 bit Weight/Act. 40+ bit Accum. | 9079 MB | 15 MB
Cheetah[18] |  HE Coeff.Pack for Conv/MatMul Output Coeff. Optim. ReLU Opt. | 37 bit Quant. | 12 MB | 3 MB
Iron[16] |  Tile for MatMul Softmax/GeLU Opt. | 37 bit Quant. | 14 MB | 3 MB
HEQuant (ours) |  Intra-Coeff.Pack for Conv/MatMul Tile for Conv Output Coeff. Optim. |  Low-Precision Quant. e.g. 4/3 bit | 4 MB | 0.2 MB

Related works There are often two categories of HE-based 2PC frameworks,
including the end-to-end HE-based frameworks [14, 32, 31, 42, 23, 11, 26, 41]
and the hybrid frameworks that combine both arithmetic secret sharing (ASS)
and HE [34, 38, 12, 37, 25]. We focus on the second category as it supports
different non-linear activations more accurately [18, 21, 16, 50, 45]. We
compare HEQuant with existing works including 2PC frameworks[46, 18, 16] and
network optimization works[6, 20, 30, 5, 35], as shown in Table 2. HEQuant
leverages an intra-coefficient packing for convolution with tiling and output
coefficient optimization. Furthermore, HEQuant utilizes low-precision network
quantization and significantly reduces both linear and non-linear layers’
communication. In addition, we summarize the notations used in the paper in
Table 1.

Figure 2: DNN private inference based on HE (linear layer).

The flow of 2PC-based inference Figure 2 shows the flow of 2PC-based
inference. With ASS, each intermediate activation tensor, e.g., $x_{i}$, is
additively shared where the server holds $\langle x_{i}\rangle^{S}$ and the
client holds $\langle x_{i}\rangle^{C}$ such that $x_{i}=\langle
x_{i}\rangle^{S}+\langle x_{i}\rangle^{C}\mod P$ [39]. For a linear layer,
e.g. Conv2D, to compute the result of $y_{i}$ without revealing both parties’
information, the client first encrypts its share as $\mathrm{Enc}(\langle
x_{i}\rangle^{C})$ and sends it to the server. Since HE supports homomorphic
addition and multiplication without the need for decryption, the server can
compute $\mathrm{Enc}(y_{i})$ and blind $y_{i}$ by subtracting a randomly
sampled $s_{i}$. Finally, the server sends the result to the client for
decryption. The server and client now obtain the secret shares of the Conv2D
output $y_{i}$111Note it is possible to move the HE computation to the pre-
processing stage [38], the required total communication remains almost the
same.. For a non-linear layer, e.g. ReLU, we omit the detailed protocols and
refer interested readers to [45, 46, 18].

At a lower level, HE encrypts the plaintexts into ciphertext polynomials and
computes over the polynomials, which can be regarded as 1-dimensional vectors.
In contrast, DNN processes high-dimensional tensors. Therefore, packing
becomes necessary to encode tensors into polynomials. Cheetah [18] identifies
that polynomial multiplications natively perform convolution operations and
introduces a polynomial coefficient packing method. In this approach, elements
in tensors are encoded into coefficients, and the correct result can be
extracted from the outcome of polynomial multiplication. A toy example is
illustrated in Figure 3.

Figure 3: A convolution example of coefficient packing.

###  2.3 Communication Complexity of HE-based 2PC

Table 3: Comm. complexity of HE-based 2PC frameworks.

Layer Type | Linear Layer (Conv.) | Non-linear Layer
---|---|---
Input Ciphertext | Output Ciphertext
Comm. Complexity | $\lceil\frac{HWC}{N}\rceil\times Nq$ | $(\lceil\frac{HWCK}{N}\rceil\times N+HWK)q$ | $\mathrm{O}(b_{x})$

The communication of a linear layer is mainly consumed by the input and output
ciphertext transfer. The communication can be accurately estimated by the
product of the number of ciphertext polynomials and the size of each
polynomial. For each polynomial, the communication is the product of the
polynomial degree and the bit width of each coefficient. Hence, the
communication of the input ciphertexts can be estimated as $\\#Poly\times Nq$.
For the output ciphertext, because each activation is encrypted into a scalar
$b$ and a polynomial $\boldsymbol{a}$, extra scalars need to be transferred.
Hence, following Cheetah [18], the output communication can be estimated with
$\\#Poly\times Nq+nq$, where $n$ is the number of output activations,
$\\#Poly\times Nq$ is the communication of $\boldsymbol{a}$, and $nq$ is the
communication of $b$. For a nonlinear layer like ReLU, the communication
scales with the bit width of activations $b_{a}$ [46]. We summarize the
communication complexity in Table 3.

Operator tiling for linear layer When the input tensor size gets larger than
the polynomial degree, it needs to be tiled into a group of small tensors
before encoding into polynomials. Operator tiling determines the strategy of
splitting the input tensors and directly impacts the communication cost [16].
For a general convolution, previous works [16, 50] focus on minimizing the
communication of input polynomials, i.e., they try to pack as many input
activations as possible. Hence, following Cheetah, given $N$, the numbers of
input polynomials and output polynomials are $\lceil\frac{HWC}{N}\rceil$ and
$\lceil\frac{HWCK}{N}\rceil$, respectively [18].

Selection of HE parameters $p$ and $q$ In HE, $p$ represents the bit width of
the plaintext field. To avoid overflow, the lower bound of $p$ is the required
accumulation bit width for a linear layer, i.e., $b_{acc}$. Given $p$, $q-p$
determines the noise budget of the HE-based computation, which is determined
by the depth of multiplications. In addition, we introduce the background of
network quantization in Appendix A.

##  3 Motivation

Figure 4: The input and output communication as well as the bit width of
plaintext and ciphertext for different bit precision quantization. Figure 5:
Overview of HEQuant and the communication/latency cost after each optimization
step. The examples are ResNet18 on CIFAR-100 and ResNet50 on ImageNet. Coeff.
represents coefficient.

In this section, we discuss the key observations that motivate our quantized
HE-based 2PC framework HEQuant. We will focus on the linear layers as they
contribute to the majority of the communication[18, 16, 37].

Observation 1: both HE parameters $p$ and $q$ can be reduced for low-precision
quantized DNN linear operators, leading to better communication efficiency. As
introduced in Section 2.3, for a linear operator, $p$ is determined by the
accumulation bit precision while $q-p$ is determined by the required noise
budget and is roughly fixed for a given operator. Low-precision quantization
helps to directly reduce the required accumulation bit precision, and thus,
improves the communication efficiency. The observation is also verified in
Figure 4. Considering the robustness of DNNs to quantization error [13, 40,
28], low-precision quantization becomes a promising technique to improve the
efficiency of the HE-based 2PC framework.

Observation 2: while networks continue to be quantized for sub-4-bit
precision, $p$ and $q$ cannot be further reduced, leading to saturated
communication efficiency improvement. Previous research has demonstrated the
feasibility of quantizing networks to sub-4 bits, such as 2-bit or even 1-bit,
while maintaining comparable accuracy [44, 43, 36, 53]. However, as shown in
Figure 4, we empirically find $p$ cannot be reduced below $13$ bits to ensure
correct computation, leading to saturated communication reduction, especially
for sub-4-bit quantization. Hence, how to further improve the communication
efficiency for ultra-low precision quantized DNNs is also an important
question to answer.

Observation 3: the input and output communication are not balanced. As shown
in Figure 4, regardless of the bit precision, the communication of output
ciphertexts is much larger compared to the input ciphertexts as more output
ciphertexts need to be transferred. Table 3 also theoretically supports it.
While past research has proposed to tile the input for matrix multiplications
and depthwise convolutions to reduce the number of output polynomials [16,
50], a general tiling algorithm has not yet been proposed for convolutions
especially when considering the impact of quantized DNNs. Moreover, as
introduced in Section 2.3, another way to reduce output communication is to
lower the coefficient bit widths. Hence, how to reduce the coefficient bit
precision beyond $q$ without impacting the output correctness is also an
important direction to study.

##  4 HEQuant: Communication-Efficient 2PC Framework

In this section, we present HEQuant, an efficient 2PC framework based on
quantization-aware HE protocols. The overview of HEQuant is shown in Figure 5.
Based on Observation 1, given a quantized network, we first propose a
quantization-aware 2PC protocol and optimize the communication by reducing the
HE parameters $p$ and $q$, achieving more than $2\times$ latency reduction
(Section 4.1). Then, to further improve the efficiency, we propose intra-
coefficient packing to allow packing multiple low-precision activations into a
single coefficient to reduce the number of input and output ciphertexts. A
series of bit width optimizations for both the plaintext and ciphertext, i.e.,
$p$ and $q$, are also developed to enable dense packing. The proposed
optimization further enables $1.4\times$ latency reduction (Section 4.2). In
the last step, a quantization-aware tiling algorithm is proposed for general
linear operations (Section 4.3). With 3-bit weight and 3-bit activation,
HEQuant achieves almost $4\times$ latency reduction over prior-art Cheetah
framework [18].

###  4.1 HE-based quantized private inference

Based on the observations discussed in Section 3, We propose the quantized HE-
based 2PC frameworks in Figure 6. We take a residual block as an example. To
avoid overflow, the HE-based convolution protocol requires the plaintext bit
width $p\geq b_{acc}$. However, the convolution input is $b_{a}$-bit. Hence,
bit extension protocols are needed to increase the bit width of activations
from $b_{a}$-bit to $b_{acc}$-bit. After the convolution, we truncate output
activations from $b_{acc}$-bit to $b_{a}$-bit to conduct low-precision ReLU to
reduce communication since the communication of ReLU scales linearly with the
activation bit width as shown in Table 3. The bit width extension, truncation,
and residual protocols are all implemented following previous work SiRNN [46].
We refer interested readers to [46] and focus on describing the quantization-
aware HE-based convolution protocol in the following sections.

###  4.2 Intra Co-efficient Packing

Figure 6: Quantized HE-based 2PC frameworks on a residual block. b_ means the
bit width for activation.  Figure 7: Examples of the proposed intra
coefficient packing.

Packing within coefficient Following Observation 2 in Section 3, the
communication reduction saturates for sub-4-bit quantized linear layers due to
the lower bound of $p$ and $q$. We observe although $p$ and $q$ cannot be
further reduced, it is possible to pack multiple low-precision activations
into a single coefficient. Consider the example in Figure 7. Two 4-bit values
are packed into one 18-bit coefficient with the two values occupying $0-4$ and
$9-13$ bits, respectively. Such packing strategy enables us to correctly
compute one multiplication with a 4-bit weight and one accumulation (which at
most requires $9$-bit accumulation) while reducing the number of coefficients
by $2\times$. Based on the observation, for a convolution, we propose two
strategies for intra-coefficient packing, namely Within-Channel Packing and
Cross-Channel Packing.

Within-Channel Packing As shown in Figure 7 (b), Within-Channel Packing
combines two activations within the same channel into one coefficient. Since
these two activations need to multiply the same weight, the weight packing
stays the same. After the homomorphic multiplication, each coefficient will
also have two output activations, resulting in a 2$\times$ reduction in the
input and output polynomials as well as the number of output coefficients.

Cross-Channel Packing Figure 7 (c) shows the Cross-Channel Packing strategy.
Two activations along the same channel are now packed into the same
coefficient. Since the two activations need to multiply different weights and
the products need to be added together, the weights packing needs to be
modified accordingly. After the homomorphic multiplication, each coefficient
only contains one output activation in higher bit locations. Hence, Cross-
Channel Packing reduces the input and output polynomials by $2\times$ while
the number of output coefficients remains unchanged.

Table 4: Comparsion of packing methods on communication and limitation as well
as the measured communication for a Conv. with dimension
(H,W,C,K,R)=(14,14,32,32,1), $b_{acc}=8$. “+” represents output coefficient
optimization. The actual value for p and q can be different for Cheetah and
our method.

Packing Method | Input Comm. | Output Comm. | Conv. Comm. | Limitation
---|---|---|---|---
Cheetah | $\lceil\frac{HWC}{N}\rceil\times Nq$ | $(\lceil\frac{HWCK}{N}\rceil\times N+HWK)q$ | 0.90 MB | -
Within-Channel | $\lceil\frac{HWC}{2N}\rceil\times Nq$ | $(\lceil\frac{HWCK}{2N}\rceil\times N+\frac{HWK}{2})q$ | 0.51 MB | kernel size=1
Cross-Channel | $\lceil\frac{HWC}{2N}\rceil\times Nq$ | $(\lceil\frac{HWCK}{2N}\rceil\times N+HWK)q$ | 0.52 MB | /
Within-Channel+ | $\lceil\frac{HWC}{2N}\rceil\times Nq$ | $\lceil\frac{HWCK}{2N}\rceil\times N(p+\sigma)+\frac{HWKp}{2}$ | 0.33 MB | kernel size=1
Cross-Channel+ | $\lceil\frac{HWC}{2N}\rceil\times Nq$ | $\lceil\frac{HWCK}{2N}\rceil\times N(\frac{p}{2}+\sigma)+HWKp$ | 0.23 MB | 1-bit error

Comparison These two packing schemes have different advantages and
disadvantages and are suitable for different situations. For one thing, Cross-
Channel Packing can be applied to convolution with any kernel size while
Within-Channel Packing can only be used for convolution with kernel size 1.
For another, they also have different communication impacts as summarized in
Table 4. As can be observed, Within-Channel Packing achieves smaller output
communication due to fewer output activations. Additionally, we observe that
in Cross-Channel Packing, only the higher bit locations of each output
activation are useful. This indicates opportunities to further optimize the
communication as will be discussed later.

Figure 8: Output coefficient optim. by discarding low-end bits.

Output coefficient optimization To further reduce the communication of output
ciphertext, we further reduce the coefficient bit widths of output
ciphertexts. Our bit width optimization is based on the following two
observations. Firstly, recall from Section 3, a $p$-bit output $m$ is
encrypted into a scalar $b$ and a polynomial $\boldsymbol{a}$. During
decryption, we compute
$m=\lfloor(b+\boldsymbol{a}^{\text{T}}\boldsymbol{s})P/Q\rfloor$ to
reconstruct the output $m$, where $Q$ and $P$ are the ciphertext modulus and
plaintext modulus, respectively. Hence, the low $q-p$ bits of each coefficient
may not be fully needed. Secondly, as mentioned above, in our Cross-Channel
Packing method, only the higher bit locations of each coefficient are useful
while the other bits may not be fully required for decryption. We illustrate
the observations in Figure 8.

Based on the observations, we propose to discard the lower bits before
ciphertext transmission. We observe discarding these lower bits is equivalent
to truncating coefficients and may introduce a single-bit error after
truncation. The error probability depends on the number of discarded bits.
Moreover, since each output activation is encrypted into a scalar $b$ and a
polynomial $\boldsymbol{a}$, how to discard $b$ or $\boldsymbol{a}$ will lead
to different error rate and communication. We empirically decide to skip the
low $\lfloor\log_{2}(Q/P)\rceil-1$ of $b$ and the low
$\lfloor\log_{2}(Q/P)\rceil+\frac{p}{2}-1-\sigma$ of $\boldsymbol{a}$ during
output ciphertext transmission, where $\sigma$ is the standard deviation of
$\boldsymbol{a}^{\text{T}}\boldsymbol{s}$. Additional details are available in
Appendix B and in the experimental results, a comprehensive ablation study
justifies this choice.

With the output coefficient optimization, the communication complexity of the
two packing methods proposed above is shown in Table 4, denoted as “Within-
Channel+” and “Cross-Channel+”.

Accumulation bit width reduction As analyzed in Section 2.3, the plaintext bit
width $p$ is selected according to the accumulation bit width $b_{acc}$.
Hence, how to reduce the accumulation bit width directly impacts the inference
cost. We adopt three strategies to estimate the required $b_{acc}$.

Strategy 1 We directly set $b_{acc}=b_{w}+b_{x}+\log_{2}(e)$ to avoid any
overflow, where $e$ is the number of accumulations. This is extensively used
in previous works [46, 2, 51, 36]. However, we find that accumulated values
rarely reach the theoretical upper bound as the weights can be either positive
or negative and most weights have small magnitude and do not reach
$2^{b_{w}}$. Hence, we propose Strategy 2 to reduce the accumulation bit
width.

Strategy 2 Considering the weights are frozen during inference, for one output
channel, we have
$\sum_{i=1}^{e}|x_{i}w_{i}|\leq\sum_{i=1}^{e}|x_{i}||w_{i}|\leq
2^{b_{x}}\sum_{i=1}^{e}|w_{i}|=2^{b_{x}}\left\|\boldsymbol{w}\right\|_{1}$. So
we just need
$b_{acc}=b_{x}+\lceil\log_{2}(\left\|\boldsymbol{w}\right\|_{1})\rceil$ to
avoid overflow. A weakness is that it may leak
$\lceil\log_{2}(\left\|\boldsymbol{w}\right\|_{1})\rceil$. However, we find
it’s still hard for the client to obtain weights.

Strategy 3 We further observe that most activations also cannot reach the
upper bound of $2^{b_{x}}$. Hence, we propose to leverage the training dataset
to directly collect the accumulation bit width required for each layer. While
this minimizes $b_{acc}$, it may introduce the overflow error if the testing
dataset does not follow the same statistics as the training dataset. We
compare the three strategies in Section 5.4 and empirically find Strategy 3
does not impact the model testing accuracy. We also show the layer-wise
accumulation bit width comparison of three strategies in Appendix C.

###  4.3 Quantization-aware operator tiling

Based on Observation 3 in Section 3, the input and output ciphertext
communication are highly imbalanced, which motivates previous works to tile
the operator to minimize the communication. Specifically, Iron and Falcon
design tiling strategies for matrix multiplication and depthwise convolution,
respectively. However, we find there exist two main drawbacks in previous
methods: 1) lack of tiling strategy for general convolution; 2) do not
consider the layerwise difference in $p,q$ induced by the network quantization
and output coefficient optimization.

To address these issues, we propose a novel quantization-aware tiling
algorithm for general convolution. We first conduct a theoretical analysis of
communication cost and formulate it as a nonlinear programming problem. Based
on Section 2.3, when introducing tiling at the channel level, we use
$C_{x},C_{w}$ to denote the number of input and weight channels in each
polynomial, respectively. Then the communication cost is given by the
equation:

$\lceil\frac{C}{C_{x}}\rceil\times Nq+(\lceil\frac{K}{C_{w}}\rceil\times
N+HWK)q$ (1)

where $\lceil\frac{C}{C_{x}}\rceil$ and $\lceil\frac{K}{C_{w}}\rceil$ are the
number of input and output polynomials, respectively. Furthermore, when output
coefficient optimization is introduced, the communication cost of Equation 1
can be replaced by that in Table 4.

Now we formulate the tiling optimization problem as a nonlinear programming
problem. Note that in Equation 1, only $C_{x},C_{w}$ needs to be determined,
with all other parameters are known before execution. Moreover, we have two
constraints: 1) polynomial coefficient encoding should not exceed the degree
$N$, and 2) $C_{x}$ must be a positive integer multiple of $C_{w}$ to avoid
doubling of the number of multiplications. Consequently, the tiling
optimization problem can be formulated as follows:

$\displaystyle\mathrm{min}\ \ \ \ $
$\displaystyle\lceil\frac{C}{C_{x}}\rceil\times
Nq+(\lceil\frac{K}{C_{w}}\rceil\times N+HWK)q$ s.t. $\displaystyle C_{x}HW\leq
N$ $\displaystyle C_{x}=kC_{w}\text{ where }k\in\mathbb{Z^{+}}$

This is a solvable nonlinear programming problem with a relatively small
solution space. With $C_{x}\leq\frac{N}{HW}$ and $C_{x}=kC_{w}$, the solution
space is at most $\frac{N}{HW}\times\frac{N}{HW}$, enabling direct solution
through a search algorithm with a complexity of $O((\frac{N}{HW})^{2})$. An
example is illustrated in Figure 9, after tiling, we reduce the size of output
polynomials.

Figure 9: A toy example of operator tiling where each square means a channel
and $W_{0,1}$ means the second channel of the first kernel of the filter.

##  5 Experimental Results

###  5.1 Experimental Setup

HEQuant is built on top of the SEAL library [47], the EMP toolkit [49] and
OpenCheetah [18] in C++. We also use the Ezpc library [4] to evaluate
CrypTFlow2 [45]. Consistent with [18, 48, 39], we simulate a LAN and WAN
network setting via Linux Traffic Control, where the bandwidth is 384 MBps and
9 MBps, respectively. All the following experiments are performed on machines
with 2.2 GHz Intel Xeon CPU. Following [16, 18], we set $N=4096$. For network
quantization, we use PyTorch framework for quantization-aware training, and
for the inference accuracy evaluation. We follow [2]’s setting to train
ResNet18 on CIFAR-100 and Tinyimagenet with different bit width, including
W8A8, W4A4, W3A3, W2A3 (WxAy means x-bit weights and y-bit activation). We
follow the setting of [51] to train ResNet50 on ImageNet with bit width W8A8,
W4A4, W3A3 and W2A2. What’s more, we use strategy3 for HEQuant to determine
the accumulation bit width. The more training details are shown in Appendix D.

###  5.2 Micro-Benchmark Evaluation

Figure 10: Latency comparison of convolutions of different dimensions (Conv
dim represents the input feature height, channels, and kernel size). DW means
depthwise convolution and Avg means the average. Figure 11: Comparison with
prior-art 2PC frameworks on three benchmarks.  Figure 12: Comparison with
prior-art network optimization algorithms on three benchmarks.

We compare the performance of HEQuant with CrypTFlow2, Iron, Cheetah, and
Falcon for single convolution and depthwise convolution with different
dimensions. We determine the accumulation bit width based on W4A4 for HEQuant.
As shown in Figure 10, HEQuant achieves on average $3.0\times$ latency
reduction compared to SOTA Iron. Note that due to the use of SIMD-based HE in
CrypTFlow2, there is an extremely high latency[18]. In comparison, HEQuant
achieves a notable latency reduction of $31\times$. We also provide the
results of communication in Appendix F.

###  5.3 End-to-End Inference Evaluation

Benchmark with prior-art 2PC frameworks We first benchmark HEQuant with prior-
art 2PC frameworks which focus on secure protocols’ improvement, including
Cheetah, Falcon, CoPriv, etc [18, 50, 54, 46, 45]. We conduct the end-to-end
private inference in our quantized networks with different bit widths. We draw
the Pareto curve of accuracy and communication/latency (LAN) in Figure 11. We
put results of latency under WAN in Appendix E.

Result and analysis From Figure 11, we make the following observations: 1)
HEQuant achieves SOTA pareto front in all three benchmarks. More specifically,
compared with Cheetah, HEQuant achieves $3.0\times$ communication and
$3.9\times$ latency reduction in ImageNet. Compared with cheetah with the same
quantization bit width, HEQuant achieves $2.0\sim 2.3\times$ communication
reduction and $1.4\sim 1.9\times$ latency reduction in three benchmarks,
respectively. Moreover, HEQuant achieves $2.3\%$ higher accuracy with less
latency in ImageNet. 2) Compared with OT-based frameworks, SiRNN and CoPriv
[54], HEQuant demonstrates strong superiority and outperform them with $52\sim
83\times$ and $91\sim 111\times$ communication reduction, respectively. This
translates to a latency improvement of an order of magnitude under WAN, as
shown in Appendix E. Since Falcon is specifically optimized for MobileNet with
depthwise convolution, we compare HEQuant with Falcon on MobileNetV2 in
Appendix G and demonstrate $2.5\times$ communication reduction.

Benchmark with prior-art network optimization algorithms Since HEQuant
requires to quantize the network, we also benchmark HEQuant with other
different network optimization algorithms, including ReLU-optimized networks
(SENet, SNL, DeepReduce, etc) and SOTA network pruning methods (MetaPruning
and uniform pruning) [30, 6, 20, 35]. We apply those algorithms to the 2PC
framework Cheetah and draw the Pareto curve in Figure 12.

Result and analysis From the results in Figure 12, we observe: 1) Compared
with ReLU-optimized networks, HEQuant achieves over $3\times$ communication by
both reducing the linear and non-linear layers’ communication; 2) Compared
with network pruning method MetaPruning, HEQuant achieves $3.6\times$
communication reduction and $4.2\times$ latency reduction as well as $1.1\%$
higher accuracy in ImageNet.

###  5.4 Ablation Study

Accumulation bit width reduction strategies In Table 5, we show the
communication of different accumulation bit width reduction strategies. We
find that even with Strategy 1, HEQuant still achieves $3.2\times$
communication reduction compared with Cheetah. With Strategy 3, HEQuant
achieves on average $3.9\times$ communication reduction over Cheetah.

Table 5: Communication (MB) comparison with different accumulation bit width
reduction strategies.

Network and Dataset | Cheetah |  HEQuant (Strategy1) |  HEQuant (Strategy2) |  HEQuant (Strategy3)
---|---|---|---|---
ResNet18 (Cifar100) | 362 | 153 $(2.4\times\downarrow)$ | 141 $(2.6\times\downarrow)$ | 102 $(3.6\times\downarrow)$
MobileNetV2 (Tinyimagenet) | 608 | 132 $(4.6\times\downarrow)$ | 123 $(4.9\times\downarrow)$ | 114 $(5.3\times\downarrow)$
ResNet50 (Imagenet) | 2454 | 771 $(3.2\times\downarrow)$ | 689 $(3.7\times\downarrow)$ | 654 $(3.8\times\downarrow)$
Average | 1141 | 352 $(3.24\times\downarrow)$ | 318 $(3.59\times\downarrow)$ | 290 $(3.94\times\downarrow)$

Figure 13: Communication and output error rate for different discarded bits
where “Discard a” represents fixing the bit width of b and discarding more
bits of $\boldsymbol{a}$.

Output coefficient optimization analysis  To show the trade-off between
communication and error rate in output coefficient optimization, we change the
number of discarded bits and profile the communication as well as the 1-bit
error rate for a convolution from ResNet50. We start with Cheetah and adopt
two strategies. The first “Discard a” means we fix the bit width of $b$ and
discard more bits of $\boldsymbol{a}$ while “Discard b” means we discard more
bits of $b$. As shown in Figure 13, 1) the communication cost decreases with
the number of discarded bits, while the error rate increases; 2) the
communication cost of “Discard a” is lower than that of “Discard b” with the
same additional discarded bits due to larger dimension of $\boldsymbol{a}$. In
addition, we find the error of “Discard a” is lower, and hence, we choose
“Discard a” as our strategy and limit the error rate below 0.0005.

Figure 14: Block-wise visualization of communication on ResNet18 with
different optimizations.

Block-wise comparison and ablation for different techniques  To understand how
different optimizations help improve communication efficiency, we add our
proposed optimizations step by step and visualize the block-wise communication
reduction on ResNet18 and present the results in Figure 14. We can see that 1)
both packing and tiling are effective and help further reduce the
communication after quantization; 2) HEQuant achieves more communication
reduction in deeper layers of the network, as $H,W$ becomes smaller and $C$
becomes larger, resulting in a larger tiling optimization space.

##  6 Conclusion

In this paper, we propose HEQuant, the first 2PC framework that marries low-
precision quantization and HE for efficient private inference. HEQuant
features an intra-coefficient packing algorithm and a quantization-aware
tiling algorithm to further reduce the communication for low-precision DNNs.
Compared with prior-art 2PC frameworks, e.g. Cheetah, HEQuant achieves
$3.0\times$ communication and $3.9\times$ latency reduction, respectively.
Compared with the prior-art network optimization algorithms, HEQuant can
achieve over $3.1\times$ communication reduction with higher accuracy.

Supplementary Material

##  Appendix A Network quantization

Quantization is one of the most impactful ways to decrease the computational
time and energy consumption of neural networks. In neural network
quantization, the weights and activation tensors are stored in lower bit
precision than the 16 or 32-bit precision they are usually trained in. When
moving from 32 to 8 bits, the memory overhead of storing tensors decreases by
a factor of 4 while the computational cost for matrix multiplication reduces
quadratically by a factor of 16. Neural networks have been shown to be robust
to quantization [40, 13], meaning they can be quantized to lower bit-widths
with a relatively small impact on the network’s accuracy. Neural network
quantization is an essential step in the model efficiency pipeline for any
practical use-case of deep learning [28, 2, 51, 36].

##  Appendix B Output coefficient optimization

Recall from Section 4.2, a $p$-bit output $m$ is encrypted into a scalar $b$
and a polynomial $\boldsymbol{a}$. During decryption, we compute
$m=\lfloor(b+\boldsymbol{a}^{\text{T}}\boldsymbol{s})P/Q\rfloor$ to
reconstruct the output $m$ and we let
$m_{e}=b+\boldsymbol{a}^{\text{T}}\boldsymbol{s}$. We observe that discarding
the lower bits of $b$ and $\boldsymbol{a}$ before ciphertext transmission can
reduce the communication cost and doesn’t affect the correctness of
decryption.

Formally, we write the high-end and low-end parts of $b$ and $\boldsymbol{a}$
as $b=b_{H}2^{l_{b}}+b_{L}$ and
$\boldsymbol{a}=\boldsymbol{a}_{H}2^{l_{a}}+\boldsymbol{a}_{L}$ such that
$0\leq b_{L}<2^{l_{b}},0\leq\boldsymbol{a}_{L}<2^{l_{a}}$. Then
$m_{e}=b_{L}+\boldsymbol{a}_{L}^{\text{T}}\boldsymbol{s}+b_{H}2^{l_{b}}+\boldsymbol{a}_{H}^{\text{T}}\boldsymbol{s}2^{l_{a}}$.
And we let $e_{L}=b_{L}+\boldsymbol{a}_{L}^{\text{T}}\boldsymbol{s}$. If we
discard the low-end parts of $b$ and $\boldsymbol{a}$, after decryption, we
will get $\lfloor(m_{e}-e_{L})P/Q\rceil=m-\lfloor e_{L}P/Q\rceil+\lfloor
eP/Q\rceil$, where $\lfloor eP/Q\rceil$ will round to 0. Remind that only the
upper half of the bits of $m$ are useful, so our goal is to ensure
$m\gg\frac{p}{2}\equiv(m-\lfloor e_{L}P/Q\rceil)\gg\frac{p}{2}$ where $\gg$
means right shift. We first approximate it by $(m-\lfloor
e_{L}P/Q\rceil)\gg\frac{p}{2}\approx(m\gg\frac{p}{2})-(\lfloor
e_{L}P/Q\rceil\gg\frac{p}{2})$. Therefore, we need to ensure $\lfloor
e_{L}P/Q\rceil\gg\frac{p}{2}=\lfloor(b_{L}+\boldsymbol{a}_{L}^{\text{T}}\boldsymbol{s})P/Q\rceil\gg\frac{p}{2}=0$.
The key here is to find out an upper bound for
$\boldsymbol{a}_{L}^{\text{T}}\boldsymbol{s}$. Since the secret vector
$\boldsymbol{s}$ distribute uniformly in $\\{0,\textpm 1\\}^{N}$ and
$\boldsymbol{a}_{L}$ distributes uniformly in $[0,2^{l_{a}})$. Then the
variance of $\boldsymbol{a}_{L}^{\text{T}}\boldsymbol{s}$ is
$\mathrm{Var}\approx\frac{2N}{9}(2^{l_{a}})^{2}$. Same with Cheetah [18], we
use $7\sqrt{\mathrm{Var}}$ as a high-probability upper bound $(\approx
1-2^{-38.5})$ on the value $\boldsymbol{a}_{L}^{\text{T}}\boldsymbol{s}$.
Finally, the constraint is
$(7\sqrt{\frac{2N}{9}}2^{l_{a}}+2^{l_{b}})P/Q<2^{\frac{p}{2}}$ and we set
$l_{b}=l^{\prime}+\frac{p}{2}-1,\
l_{a}=l^{\prime}+\frac{p}{2}-1-\lfloor\log_{2}(7\sqrt{\frac{2N}{9}})\rfloor$
to meet the constraint, where $l^{\prime}=\lfloor\log_{2}(Q/P)\rceil$.

However, the approximation of $(m-\lfloor
e_{L}P/Q\rceil)\gg\frac{p}{2}\approx(m\gg\frac{p}{2})-(\lfloor
e_{L}P/Q\rceil\gg\frac{p}{2})$ may cause least significant bit (LSB) error. To
be more specific, there will be a $1/(2^{q-p/2-\log_{2}(e_{L})})$ probability
of causing a 1-bit LSB error. Therefore, we decrease $e_{L}$ with several
tricks to reduce the probability of LSB error. The first trick is to decrease
the number of discarded bits of $b$ rather than $\boldsymbol{a}$ because
$\boldsymbol{a}$ accounts for larger communication. Moreover, after each
linear layer, we need to truncate the output to low bit width $b_{a}$ where
the output will be divided by a scaling factor. That means more low-end bits
of the output are useless and can help us reduce the 1-bit error rate.
Ultimately, we make the error rate lower than 0.0005 by skipping the low
$l^{\prime}-1$ of $b$ and the low $l^{\prime}+\frac{p}{2}-1-\sigma$ of
$\boldsymbol{a}$ where $\sigma=\lfloor\log_{2}(7\sqrt{\frac{2N}{9}})\rfloor$.

##  Appendix C Visualization of different accumulation bit width reduction
strategies

Figure 15: Layer-wise accumulation bit width in ResNet32 with 4-bit for both
weight and activation.

Figure 15 visualizes the layer-wise accumulation bit width in a ResNet18
network trained on the CIFAR-100 dataset with different strategies proposed in
Section 4.2.

##  Appendix D Experiment setting

###  D.1 Network architecture

Table 6: HEQuant evaluation benchmarks.

Model | Layers | # Params (M) | MACs (G) | Dataset
---|---|---|---|---
ResNet18 | 20 Conv, 1 FC, 1 AP, 17 ReLU | 11.22 | 0.558 | CIFAR-100/Tinyimagenet
ResNet50 | 49 CONV, 1 FC, 1, MP, 1 AP, 49 ReLU | 2.5 | 4.1 | Imagenet

We evaluate and compare HEQuant with prior-art methods on three networks on
three datasets, and the details of which are shown in Table 6.

###  D.2 Training details

We train ResNet18 following the training scheme in [2] to use real-valued
PyTorch pre-trained models as initialization for corresponding quantized
networks. We use Adam optimizer [27] with a cosine learning rate decay
scheduler. The initial learning rate is set to $55e-5$ for weight parameters
and batch size is set to $256$. We set weight decay to be $6e-2$. The models
are trained for $400$ epochs without a knowledge distillation scheme. We adopt
the basic data augmentation as ResNet [17]. Training images are randomly
resized and cropped to $32\times 32$ pixels ($64\times 64$ pixels for
Tinyimagenet) and randomly flipped horizontally. Test images are center-
cropped to the same resolution.

For ResNet50, we follow [51] with the same training settings. Specifically,
with an initial learning rate of 0.1 for the weights and an initial learning
rate of 0.01 for the clipping and companding parameters, the models were
trained over 120 epochs with a mini-batch size of 512 for ResNet50. In
addition, we applied a warm-up method for the first 5 epochs and increased the
learning rate linearly from $10^{-4}$ to the initial value. The weight decay
was set to $4\times 10^{-5}$. The training images were resized, cropped to
$224\times 224$ pixels and randomly flipped horizontally. The test images were
center-cropped to $224\times 224$ pixels.

##  Appendix E Experimental results of latency

Figure 17 and Figure 18 show the latency comparison with prior-art 2PC
frameworks and network optimization algorithms on three benchmarks under LAN
and WAN. We can see that: 1): HEQuant achieves over $3\times$ latency
reduction compared with prior-art 2PC frameworks and over $2.5\times$ latency
reduction compared with prior-art network optimization algorithms; 2) Under
WAN, HEQuant demonstrates more significant latency reduction than under LAN
due to the large communication reduction achieves by HEQuant.

##  Appendix F Communication comparison on Micro-Benchmark

Same with Section 5.2, we compare the communication cost of HEQuant with
CrypTFlow2, Iron, Cheetah, and Falcon for single convolution and depthwise
convolution with different dimensions. As shown in Figure 16, HEQuant achieves
on average $3.0\times$ communication reduction compared to SOTA Iron. Note
that the communication cost of CrypTFlow2 is relatively low. However,
CrypTFlow2 uses SIME-based HE and suffers from extremely high latency as shown
in Section 5.2.

##  Appendix G Comparison with Falcon

We compare HEQuant with Falcon on MobileNetV2 in Table 7. Though Falcon is
optimized specifically for MobileNetV2, HEQuant still achieves $2.5\times$
communication and $1.9\times$ latency reduction.

Table 7: Comparison with Falon on MobileNetV2 on the TinyImagenet dataset with
W4A4.

MobileNetV2 | Comm. (MB) | Latency (s)
---|---|---
Cheetah |  561.7 $(4.8\times)$ |  86.3 $(3.0\times)$
Falcon |  296.9 $(2.5\times)$ |  55.9 $(1.9\times)$
HEQuant | 118.3 | 29.0

Figure 16: Communication comparison of convolutions of different dimensions
(Conv dim represents the input feature height, channels, and kernel size). DW
means depthwise convolution and Avg means the average. Figure 17: Latency
comparison with prior-art 2PC frameworks on three benchmarks under LAN and
WAN. Figure 18: Latency comparison with prior-art network optimization
algorithms on three benchmarks under LAN and WAN.

## References

  * Azouji et al. [2022] Neda Azouji, Ashkan Sami, and Mohammad Taheri.  Efficientmask-net for face authentication in the era of covid-19 pandemic.  _Signal, Image and Video Processing_ , 16(7):1991–1999, 2022.
  * Bhalgat et al. [2020] Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak.  Lsq+: Improving low-bit quantization through learnable offsets and better initialization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 696–697, 2020.
  * Brill et al. [2022] Thomas M Brill, Laura Munoz, and Richard J Miller.  Siri, alexa, and other digital assistants: a study of customer satisfaction with artificial intelligence applications.  In _The Role of Smart Technologies in Decision Making_ , pages 35–70. Routledge, 2022.
  * Chandran et al. [2017] Nishanth Chandran, Divya Gupta, Aseem Rastogi, Rahul Sharma, and Shardul Tripathi.  Ezpc: Programmable, efficient, and scalable secure two-party computation for machine learning.  _Cryptology ePrint Archive_ , 2017.
  * Cho et al. [2022a] Minsu Cho, Zahra Ghodsi, Brandon Reagen, Siddharth Garg, and Chinmay Hegde.  Sphynx: A deep neural network design for private inference.  _IEEE Security & Privacy_, 20(5):22–34, 2022a.
  * Cho et al. [2022b] Minsu Cho, Ameya Joshi, Brandon Reagen, Siddharth Garg, and Chinmay Hegde.  Selective network linearization for efficient private inference.  In _International Conference on Machine Learning_ , pages 3947–3961. PMLR, 2022b.
  * Choi et al. [2022] Woo-Seok Choi, Brandon Reagen, Gu-Yeon Wei, and David Brooks.  Impala: Low-latency, communication-efficient private deep learning inference, 2022.
  * Dathathri et al. [2019] Roshan Dathathri, Olli Saarikivi, Hao Chen, Kim Laine, Kristin Lauter, Saeed Maleki, Madanlal Musuvathi, and Todd Mytkowicz.  Chet: an optimizing compiler for fully-homomorphic neural-network inferencing.  In _Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation_ , 2019.
  * Demmler et al. [2015] Daniel Demmler, Thomas Schneider, and Michael Zohner.  Aby -a framework for efficient mixed-protocol secure two-party computation.  In _Proceedings 2015 Network and Distributed System Security Symposium_ , 2015.
  * Dong et al. [2020] Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.  Hawq-v2: Hessian aware trace-weighted quantization of neural networks.  _Advances in neural information processing systems_ , 33:18518–18529, 2020.
  * Fan et al. [2023] Shengyu Fan, Zhiwei Wang, Weizhi Xu, Rui Hou, Dan Meng, and Mingzhe Zhang.  Tensorfhe: Achieving practical computation on encrypted data using gpgpu.  In _2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_ , pages 922–934. IEEE, 2023.
  * Garimella et al. [2022] Karthik Garimella, Zahra Ghodsi, NandanKumar Jha, Siddharth Garg, and Brandon Reagen.  Characterizing and optimizing end-to-end systems for private inference, 2022.
  * Gholami et al. [2022] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer.  A survey of quantization methods for efficient neural network inference.  In _Low-Power Computer Vision_ , pages 291–326. Chapman and Hall/CRC, 2022.
  * Gilad-Bachrach et al. [2016] Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing.  Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy.  In _International conference on machine learning_ , pages 201–210. PMLR, 2016.
  * Gupta et al. [2022] Kanav Gupta, Deepak Kumaraswamy, Nishanth Chandran, and Divya Gupta.  Llama: A low latency math library for secure inference.  _Proceedings on Privacy Enhancing Technologies_ , 2022(4):274–294, 2022.
  * Hao et al. [2022] Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang.  Iron: Private inference on transformers.  In _Advances in Neural Information Processing Systems_ , 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Huang et al. [2022] Zhicong Huang, Wen-jie Lu, Cheng Hong, and Jiansheng Ding.  Cheetah: Lean and fast secure $\\{$Two-Party$\\}$ deep neural network inference.  In _31st USENIX Security Symposium (USENIX Security 22)_ , pages 809–826, 2022.
  * Hussain et al. [2021] Siam Umar Hussain, Mojan Javaheripi, Mohammad Samragh, and Farinaz Koushanfar.  Coinn: Crypto/ml codesign for oblivious inference via neural networks.  In _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_ , pages 3266–3281, 2021.
  * Jha et al. [2021] Nandan Kumar Jha, Zahra Ghodsi, Siddharth Garg, and Brandon Reagen.  Deepreduce: Relu reduction for fast private inference.  In _International Conference on Machine Learning_ , pages 4839–4849. PMLR, 2021.
  * Juvekar et al. [2018] Chiraag Juvekar, Vinod Vaikuntanathan, and AnanthaP. Chandrakasan.  GAZELLE: A low latency framework for secure neural network inference, 2018.
  * Kaissis et al. [2021] Georgios Kaissis, Alexander Ziller, Jonathan Passerat-Palmbach, Théo Ryffel, Dmitrii Usynin, Andrew Trask, Ionésio Lima Jr, Jason Mancuso, Friederike Jungmann, Marc-Matthias Steinborn, et al.  End-to-end privacy preserving deep learning on multi-institutional medical imaging.  _Nature Machine Intelligence_ , 3(6):473–484, 2021.
  * Kim and Guyot [2023] Dongwoo Kim and Cyril Guyot.  Optimized privacy-preserving cnn inference with fully homomorphic encryption.  _IEEE Transactions on Information Forensics and Security_ , 18:2175–2187, 2023.
  * Kim et al. [2023a] Donghwan Kim, Jaiyoung Park, Jongmin Kim, Sangpyo Kim, and JungHo Ahn.  Hyphen: A hybrid packing method and optimizations for homomorphic encryption-based neural networks, 2023a.
  * Kim et al. [2023b] Donghwan Kim, Jaiyoung Park, Jongmin Kim, Sangpyo Kim, and Jung Ho Ahn.  Hyphen: A hybrid packing method and optimizations for homomorphic encryption-based neural networks.  _arXiv preprint arXiv:2302.02407_ , 2023b.
  * Kim et al. [2022] Jongmin Kim, Gwangho Lee, Sangpyo Kim, Gina Sohn, Minsoo Rhu, John Kim, and Jung Ho Ahn.  Ark: Fully homomorphic encryption accelerator with runtime data generation and inter-operation key reuse.  In _2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO)_ , pages 1237–1254. IEEE, 2022.
  * Kingma and Ba [2014] DiederikP. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv: Learning,arXiv: Learning_ , 2014.
  * Krishnamoorthi [2018] Raghuraman Krishnamoorthi.  Quantizing deep convolutional networks for efficient inference: A whitepaper.  _arXiv preprint arXiv:1806.08342_ , 2018.
  * Kumar et al. [2020] Nishant Kumar, Mayank Rathee, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma.  Cryptflow: Secure tensorflow inference.  In _2020 IEEE Symposium on Security and Privacy (SP)_ , pages 336–353. IEEE, 2020.
  * Kundu et al. [2023] Souvik Kundu, Shunlin Lu, Yuke Zhang, Jacqueline Liu, and Peter A Beerel.  Learning to linearize deep neural networks for secure and efficient private inference.  _arXiv preprint arXiv:2301.09254_ , 2023.
  * Lee et al. [2022a] Eunsang Lee, Joon-Woo Lee, Junghyun Lee, Young-Sik Kim, Yongjune Kim, Jong-Seon No, and Woosuk Choi.  Low-complexity deep convolutional neural networks on fully homomorphic encryption using multiplexed parallel convolutions.  In _International Conference on Machine Learning_ , pages 12403–12422. PMLR, 2022a.
  * Lee et al. [2022b] Joon-Woo Lee, HyungChul Kang, Yongwoo Lee, Woosuk Choi, Jieun Eom, Maxim Deryabin, Eunsang Lee, Junghyun Lee, Donghoon Yoo, Young-Sik Kim, et al.  Privacy-preserving machine learning with fully homomorphic encryption for deep neural network.  _IEEE Access_ , 10:30039–30054, 2022b.
  * Lehmkuhl et al. [2021] Ryan Lehmkuhl, Pratyush Mishra, Akshayaram Srinivasan, and Raluca Ada Popa.  Muse: Secure inference resilient to malicious clients.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pages 2201–2218, 2021.
  * Liu et al. [2017] Jian Liu, Mika Juuti, Yao Lu, and N. Asokan.  Oblivious neural network predictions via minionn transformations.  In _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ , 2017.
  * Liu et al. [2019] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun.  Metapruning: Meta learning for automatic neural network channel pruning.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 3296–3305, 2019.
  * Liu et al. [2022] Zechun Liu, Kwang-Ting Cheng, Dong Huang, Eric P Xing, and Zhiqiang Shen.  Nonuniform-to-uniform quantization: Towards accurate quantization via generalized straight-through estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4942–4952, 2022.
  * Lu et al. [2023] Wen-jie Lu, Zhicong Huang, Zhen Gu, Jingyu Li, Jian Liu, Kui Ren, Cheng Hong, Tao Wei, and WenGuang Chen.  Bumblebee: Secure two-party inference framework for large transformers.  _Cryptology ePrint Archive_ , 2023.
  * Mishra et al. [2020] Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and RalucaAda Popa.  Delphi: A cryptographic inference service for neural networks, 2020.
  * Mohassel and Zhang [2017] Payman Mohassel and Yupeng Zhang.  Secureml: A system for scalable privacy-preserving machine learning.  In _2017 IEEE symposium on security and privacy (SP)_ , pages 19–38. IEEE, 2017.
  * Nagel et al. [2021] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort.  A white paper on neural network quantization.  _arXiv preprint arXiv:2106.08295_ , 2021.
  * Onoufriou et al. [2021] George Onoufriou, Paul Mayfield, and Georgios Leontidis.  Fully homomorphically encrypted deep learning as a service.  _Machine Learning and Knowledge Extraction_ , 3(4):819–834, 2021.
  * Park et al. [2023] Jaiyoung Park, Donghwan Kim, Jongmin Kim, Sangpyo Kim, Wonkyung Jung, Jung Hee Cheon, and Jung Ho Ahn.  Toward practical privacy-preserving convolutional neural networks exploiting fully homomorphic encryption.  _arXiv preprint arXiv:2310.16530_ , 2023.
  * Qin et al. [2020a] Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe.  Binary neural networks: A survey.  _Pattern Recognition_ , 105:107281, 2020a.
  * Qin et al. [2020b] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song.  Forward and backward information retention for accurate binary neural networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2250–2259, 2020b.
  * Rathee et al. [2020] Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma.  Cryptflow2: Practical 2-party secure inference.  In _Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security_ , pages 325–342, 2020.
  * Rathee et al. [2021] Deevashwer Rathee, Mayank Rathee, Rahul Kranti Kiran Goli, Divya Gupta, Rahul Sharma, Nishanth Chandran, and Aseem Rastogi.  Sirnn: A math library for secure rnn inference.  In _2021 IEEE Symposium on Security and Privacy (SP)_ , pages 1003–1020. IEEE, 2021.
  * [47] SEAL.  Microsoft SEAL (release 3.6).  https://github.com/Microsoft/SEAL, 2020.  Microsoft Research, Redmond, WA.
  * Shen et al. [2022] Liyan Shen, Ye Dong, Binxing Fang, Jinqiao Shi, Xuebin Wang, Shengli Pan, and Ruisheng Shi.  Abnn 2.  In _Proceedings of the 59th ACM/IEEE Design Automation Conference_ , 2022.
  * Wang et al. [2016] Xiao Wang, Alex J. Malozemoff, and Jonathan Katz.  EMP-toolkit: Efficient MultiParty computation toolkit.  https://github.com/emp-toolkit, 2016.
  * Xu et al. [2023] Tianshi Xu, Meng Li, Runsheng Wang, and Ru Huang.  Falcon: Accelerating homomorphically encrypted convolutions for efficient private mobile network inference.  _arXiv preprint arXiv:2308.13189_ , 2023.
  * Yamamoto [2021] Kohei Yamamoto.  Learnable companding quantization for accurate low-bit neural networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5029–5038, 2021.
  * Yao et al. [2021] Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael Mahoney, et al.  Hawq-v3: Dyadic neural network quantization.  In _International Conference on Machine Learning_ , pages 11875–11886. PMLR, 2021.
  * Yuan and Agaian [2023] Chunyu Yuan and Sos S Agaian.  A comprehensive review of binary neural network.  _Artificial Intelligence Review_ , pages 1–65, 2023.
  * Zeng et al. [2023] Wenxuan Zeng, Meng Li, Haichuan Yang, Wen-jie Lu, Runsheng Wang, and Ru Huang.  Copriv: Network/protocol co-optimization for communication-efficient private inference.  _arXiv preprint arXiv:2311.01737_ , 2023.