*[inlinelist,1]label=(), marginparsep has been altered.
topmargin has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

Towards Efficient and Scalable Training of Differentially Private Deep
Learning

Sebastian Rodriguez Beltran 1  Marlon Tobaben 1  Niki Loppi 2  Antti Honkela 1

††footnotetext:  1Department of Computer Science, University of Helsinki,
Helsinki, Finland 2NVIDIA, Helsinki, Finland. Correspondence to: Sebastian
Rodriguez Beltran<EMAIL_ADDRESS>Marlon Tobaben
<EMAIL_ADDRESS>Niki Loppi<EMAIL_ADDRESS>Antti Honkela
<EMAIL_ADDRESS>
Accepted to the Workshop on Advancing Neural Network Training at International
Conference on Machine Learning (WANT@ICML 2024).

###### Abstract

Differentially private stochastic gradient descent (DP-SGD) is the standard
algorithm for training machine learning models under differential privacy
(DP). The major drawback of DP-SGD is the drop in utility which prior work has
comprehensively studied. However, in practice another major drawback that
hinders the large-scale deployment it is the significantly higher
computational cost. We conduct a comprehensive empirical study to quantify the
computational cost of training deep learning models under DP and benchmark
methods that aim at reducing the cost. Among these are more efficient
implementations of DP-SGD and training with lower precision. Finally, we study
the scaling behaviour using up to 80 GPUs.

##  1 Introduction

Training data of machine learning (ML) models can be extracted Balle et al.
(2022); Carlini et al. (2021). Differential Privacy (DP) (Dwork et al., 2006)
is the gold standard for formalizing the privacy leakage of training data in
ML and mitigating the risk of privacy attacks on the training data. DP is
deployed in many applications involving sensitive data Abowd (2018); Cormode
et al. (2018).

Figure 1: Relative throughput to the respective non private baseline. For each
optimization method and each model size, we divide its throughput with the
non-private counterpart. Throughput is the number of processed instances per
second. The optimization mode includes clipping optimizations, lower
precision, and frameworks like JAX.

The established algorithm for integrating DP into the training pipeline of
deep learning models is DP stochastic gradient descent (DP-SGD) Rajkumar &
Agarwal (2012); Song et al. (2013); Abadi et al. (2016), which is the DP
adaptation of SGD (see also Algorithm 1). DP-SGD has two major drawbacks in
comparison to SGD:

  * •

Higher computational cost: DP-SGD requires more memory and is computationally
more expensive due to the per-sample clipping.

  * •

Loss in utility: The utility in comparison to non-DP training drops but this
can be mitigated to certain extend by using larger batch sizes (Räisä et al.,
2024) and training longer Ponomareva et al. (2023) which further increases the
computational cost.

List of contributions In this work we perform an extensive empirical study on
the computational efficiency of DP-SGD. We will focus on fine-tuning a wide
range of large computer vision classification models but our findings can be
applied any other large models that are trained or fine-tuned with DP-SGD. Our
main contributions are the following:

  1. 1.

We find that non-optimized training with DP-SGD costs up to three and eight
times more than non-private training for ViT and ResNets, respectively (See
Section 4).

  2. 2.

We identify the reasons that lead to the higher computational cost of DP-SGD
using profiling (See Section 4.3).

  3. 3.

We benchmark different strategies that can reduce this cost drastically up to
a level that matches the non-optimized non-private training:

     * •

More efficient implementations of DP-SGD (See Section 5.1).

     * •

Use of JAX instead of PyTorch (See Section 5.2).

     * •

Lower Precision with TF-32 (See Section 5.3).

  4. 4.

We scale up the training to 80 GPUs and find that DP-SGD scales better than
non-private training (See Section 6).

##  2 Background

This section will explain the main DP-SGD algorithm and optimizations to
alleviate its computational cost.

###  2.1 DP-SGD Algorithm

Algorithm 1 is the original DP-SGD algorithm, with virtual batching, as
proposed by Abadi et al. (2016).

Algorithm 1 Virtual Batching DP-SGD

Input: Training data points $\left\\{x_{1},\dots,x_{N}\right\\}$, loss
function $\mathcal{L}(\theta)=\frac{1}{N}\sum_{i}\mathcal{L}(\theta,x_{i})$

Parameters: Parameters: learning rate $\eta_{t}$, noise scale $\sigma$,
gradient norm bound $C$, number of steps $T$, logical batch size $L$, physical
batch size $p$.

for $t\in[T]$ do

B $\leftarrow$ Sample batch by Poisson sampling $L/N$.

P $\leftarrow$ Divide the logical batch B into physical batches of size $p$.

$\theta_{acc}\leftarrow 0$

for $s\in[P]$ do

Compute gradient

For each $i\in s$ compute
$\textbf{g}_{t}(x_{i})\leftarrow\nabla_{\theta_{t}}\mathcal{L}(\theta_{t},x_{i})$

Clip gradient

$\overline{\mbox{{g}}}_{t}(x_{i})\leftarrow\textbf{g}_{t}/\text{max}(1,\frac{\|\textbf{g}_{t}(x_{i})\|_{2}}{C})$

Accumulate gradient

$\theta_{acc}\leftarrow\theta_{acc}+\sum_{i}\overline{\mbox{{g}}}_{t}(x_{i})$

end for

Add noise

$\widetilde{\textbf{g}}_{t}\leftarrow\frac{1}{|L|}(\theta_{acc}+\mathcal{N}(0,\sigma^{2}C^{2}\textbf{I}))$

Step

$\theta_{t+1}\leftarrow\theta_{t}-\eta_{t}\widetilde{\textbf{g}}_{t}$

end for

Return Learned parameters $\theta_{T}$ and then the privacy cost is computed.

Virtual Batching distinguishes between logical and physical batches. Logical
batches are divided into multiple physical batches to allow taking optimizer
steps with many samples without running out of memory. In our experiments, we
typically sample logical batch sizes of size $L=25000$ while the memory only
fits $<300$ samples at a time. Implementing DP-SGD with virtual batching
Algorithm 1 does not modify the privacy accounting. The amount of added noise
is the same and does not affect the model utility (Ponomareva et al., 2023).

Opacus Our baseline uses the PyTorch (Ansel et al., 2024) library Opacus
(Yousefpour et al., 2021). Opacus is the most mature DP-SGD framework out of
all considered implementations. It supports nearly all neural layers that are
compatible with DP training. Opacus implements per-sample clipping without any
additional optimizations. Opacus implements virtual batching algorithm in
their `BatchMemoryManager`. The privacy engine of Opacus will sample the
logical batches, as in the original DP-SGD algorithm, and then divide them
into physical batches. The other implementations considered in our experiments
do not support virtual batching out-of-the-box.

Poisson subsampling Interestingly, Bu et al. (2022) and Bu et al. (2023) never
mention Poisson subsampling in their works of Mix Ghost Clipping and Book
Keeping. Even more, Bu et al. (2022) states that it has a speed-up of $\times
1.7$ times against other algorithms with a fixed batch size, which would
affect the privacy accountant method. The same happens in practice for JAX
implementations De et al. (2022), were the sampling is done by shuffling the
dataset and using each sample once per epoch. While it is easier to implement
in practice, it does not use the correct Poisson subsampling for the numerical
accounting methods.

To make a fair comparison between all methods, we implement the Poisson
subsampling, the same way Opacus does it, for all frameworks and a new custom
Batch Memory Manager to flag when it is time to take a step. This way, all the
experiments are seeded and will have the same logical and physical batch
sizes.

###  2.2 DP-SGD Optimizations

We benchmark five types of clipping methods. Table A1 shows which clipping
optimizations we are benchmark against the library or framework that
implements it.

JAX We compare all implementations with a native JAX Bradbury et al. (2018)
implementation that clips the per-sample gradients with Optax DeepMind et al.
(2020) without utilizing any further optimization.

Ghost clipping computes the gradient norm loss after the backpropagation
optimization and then reweights the loss to update the clipped gradients. Its
main contribution is memory saving at the cost of adding another backward pass
(Li et al., 2022).

Mixed Ghost clipping proposed by Bu et al. (2022) builds on-top of Ghost
clipping. It implements the ghost clipping technique for convolutional layers.
Its main contribution is that the algorithm will decide when to clip the
gradients using per-example or ghost. This difference matters because the
ghost clipping is less efficient when the layer’s input dimensions are too
big. E.g., for ResNets, each clipping method will be applied for half of the
layers. The first layers will be clipped using the the per-example and then
ghost clipping in the bottom layers. As the model goes deeper, the feature
size decreases, and the number of channels increases, prioritizing ghost
clipping (Bu et al., 2022).

Book keeping by Bu et al. (2023) uses all the previous techniques but requires
only one backpropagation pass without explicitly calculating the per-example
gradients. It avoids the second pass that ghost clipping does by reusing the
intermediate results of the output gradients to calculate the sum of the
clipped gradients and the clipping factor. Book keeping can also be
implemented together with the mix optimization. It also implements another
technique called MixOpt, which does the same evaluation as the mix ghost
clipping, but also determines whether doing a second backward pass is more
efficient.

Implementations of optimized DP-SGD PrivateVision (Bu et al., 2022), and
FastDP (Bu et al., 2023) are PyTorch-based implementations. While both
PrivateVision and FastDP implement ghost clipping and its variants, all FastDP
implementations use the book keeping method.

##  3 Experiment Overview

In our experiments we compare the throughput, defined as how many samples can
be processed per second during training.

Datasets We benchmark with the dataset CIFAR100 (Krizhevsky & Hinton, 2009).
It contains 60,000 data points, from 100 classes. It is a commonly used
dataset for testing computer vision models and their private counterparts
(Abadi et al., 2016; De et al., 2022; Yousefpour et al., 2021).

Models We benchmark two families of models. The Vision Transformer (ViT)
(Dosovitskiy et al., 2021) and BiT-ResNet (Kolesnikov et al., 2020) (SeeTable
1). All models are pre-trained on the same public dataset, ImageNet-21k.
However, the ViT models are additionally fine-tuned on ImageNet-1k (Steiner et
al., 2022; Wightman, 2019). For our experiments, we are fine-tuning the
feature encoder on CIFAR100. CIFAR-100 images are resized to 224x224.

Parameterization While parameter-efficient fine-tuning of some parts of the
model, e.g. using adapters like FiLM Perez et al. (2018) or LoRA (Hu et al.,
2022) has been shown to be effective under DP Tobaben et al. (2023); Yu et al.
(2022), our work focuses on the computational efficiency of DP-SGD and thus we
consider on the worst-case scenario which is fine-tuning all parameters of the
model. Furthermore, any training from scratch requires training all
parameters.

Table 1: Number of parameters, in millions, for each family architecture and
size of the model.
Model | # of Parameters
---|---
ViT | Tiny | 5.7 M
Small | 22.1 M
Base | 86.6 M
Large | 304.3 M
Huge | 630.8 M
BiT-ResNet | $50\times 1$ | 23.7 M
$101\times 1$ | 42.7 M
$50\times 3$ | 211.8 M
$101\times 3$ | 382.4 M
$152\times 4$ | 929.2 M

Hyperparameters All experiments use the optimal hyperparameters found
previously by Tobaben et al. (2023) with a privacy budget of $\epsilon=8$. We
train for two epochs, and according to the sampling rate and batch sizes,
there will be four optimizer steps in total. Using only two epochs allows us
to test the experiments quickly and evaluate them over multiple runs. We do
not focus on finding the best possible utility, which requires training for
many more epochs. In Table A4 we show the mean accuracy after training for
just two epochs. For optimal utility, it must be trained for a higher number
of epochs.

Environment specifications We use two GPU architectures, NVIDIA V100 and
NVIDIA A100, with 32 and 40 GB of VRAM respectively. Both use identical Python
environments, duplicated in both to make a fair comparison. For the multinode
experiments, each node has four GPU units. All experiments use 16 CPU workers,
except the distributed DP, which breaks with more than one worker.

Source code We will make the source code for replicating the experiments
available at https://github.com/DPBayes/Towards-Efficient-Scalable-Training-
DP-DL.

##  4 What is the cost of DP in Deep Learning?

In this section we will quantify the computational cost of deploying DP
training as a whole. We do this by comparing the maximum physical batch sizes
and throughputs between the non-private training and Opacus per-example
clipping, called O-flat, which is the most used DP-SGD implementation.
Additionally, we identify the reasons for the higher computational cost of DP-
SGD through profiling.

###  4.1 Maximum Physical Batch Size

(a) ViT Models

(b) BiT ResNet Models

Figure 2: Maximum achievable physical batch size by the different model sizes
on A100 GPU (40 GB) before they reach Out Of Memory (OOM) Error. The model
sizes grow from left to right. To check the number of parameters of each size,
refer to Table 1.

We first compare how many samples we can process in one physical batch before
running out of memory. In Figure 2, we can see that the model size matters as
its parameters must be stored in memory, and their gradients must also be
stored for accumulation. As there are millions of parameters, it becomes
increasingly expensive to store them in memory. As expected, due to its
implementation of calculating the gradients for each sample, the per-example
clipping fits fewer instances.

The difference in throughput between non-private and private training grows
with respect to the model size. In Figure 2(a), the difference for the huge
model size is already 11x times the number of fitted samples, and the private
model can only fit three samples, getting closer to just calculating one
sample at a time and accumulating that for a large logical batch, which is
very inefficient. This is problematic as a bigger model would be unable to fit
even one sample.

From the experiments we also verified that the use of physical batches does
not affect the model performance; each model size will reach a different
accuracy than the others, but the same model size will always reach the same
performance independent of the physical batch size.

###  4.2 Throughput Comparision

The batch size alone does not tell the whole story, as the main interest is
how much slower the private training is. Figure 3 shows the relative
difference between mean throughput, which is how faster the non-private
training is and how much more expensive the private one is. It is consistent
with Figure 2 in that as the model size grows, the difference between how many
samples can be fitted and the throughput difference will grow larger.

ViT The throughput difference between Opacus and a non-private baseline for
the ViT model (see Figure 3(a)) does not spike and grows steadily, which is
interesting considering how big the relative difference is in physical batch
size. Using the previous example with the ViT huge model, the throughput
difference is about $\times 3.1$, while we can fit 11 times more samples in
the non-private implementation.

ResNets On the other hand, in Figure 3(b), the ResNet models have different
behavior. They do have spikes of growth as the model size grows. The contrast
in Figure 3 between ViT and BiT ResNet models is due to the architecture and
types of layers. The parameter space grows as the width factor (see Table 1)
for the ResNets, so the $\times 3$ makes the neural network wider by a factor
of three. Based on our results, the width of the layers affects throughput
much more than the depth of the network. They have comparable throughput with
the same width and different depths, but increasing the width will make the
model in the private setting much slower and reduce the maximum batch size
significantly.

(a) ViT Models

(b) BiT ResNet Models

Figure 3: Relative difference between mean throughputs between Opacus per-
example clipping and the non-private baseline. It is defined as private-
throughput/non-private-throughput. It shows how many times private training is
more expensive. These experiments are executed on one A100 GPU.

How much does finding the maximum physical batch size matter? In Figure 4, we
display the relative throughput as a percentage by dividing the throughput at
a particular physical batch size by the maximum achievable throughput. We see
that as the physical batch size increases, the throughput will grow as
expected, but at some point there is no signifant further improvement in
throughput from using a larger physical batch size.

Figure 4: The relative difference with the throughput at the maximum batch
size for the ViT base model on A100.

Practitioners may estimate the optimal batch size based on available memory
and performance trade-offs. It is not crutial to set the physical batch size
to the maximum possible but a good enough value be fine. Typically, smaller
batches are limited by data loading speeds, but as batch size increases,
computation becomes the limiting factor.

###  4.3 Reasons for the Increase in Computational Cost

Giving a detailed breakdown of low-level operations associated with DP is
challenging. However, using GPU profiling tool NVIDIA Nsight System we can
identify three aspects which constitute to the majority DP overheads. Firstly,
due to its larger memory footprint, DP-SGD is able to consume smaller physical
batches than its non-private counterpart. This results in larger amount of
smaller low-level kernel calls which leads to slightly lower utilisation of
the GPU compute. At very small batch sizes even the kernel launch overheads
can become a notable factor for slowdown. Secondly, the computation of per-
example-gradients introduces significant overhead in the backward pass as it
cannot be parallelised as in batched gradient computation. This is the most
prominent cause of the total overhead. Finally, an additional DP-optimizer
step that clips and accumulates the per example gradients, which is not
present in the non-dp algorithm, needs to taken after each physical batch (see
Table A3).

##  5 Decreasing the computational cost

This section analyzes the different strategies for training with DP-SGD more
efficiently. We evaluate both algorithmic and hardware optimizations and their
combinations.

###  5.1 Optimized algorithms

First, we evaluate the more efficient implementations that have been described
in Section 2.2 using the ViT base model. We chose it as our benchmark model
because the middle model size is large enough to test our hypothesis but less
extensive than it would take too much time to train. The implementations do
not support the BiT ResNet due to their custom weight standardization layer.

Table 2: Maximum physical batch size reachable for each clipping method, for
the two GPU architectures we are comparing, for the ViT base model.
Clipping Mode | V100 | A100
---|---|---
Non private baseline | 216 | 268
O-flat (Opacus) | 28 | 35
Ghost (Private Vision) | 203 | 257
Mix Ghost (Private Vision) | 203 | 257
BK Ghost (FastDP) | 189 | 209
BK Mix Ghost (FastDP) | 189 | 209
BK Mix Opt (FastDP) | 189 | 209

Maximum physical batch size Table 2 compares the maximum physical batch size
for both available GPUs. The maximum physical batch size is larger for the
optimizations of DP-SGD than for Opacus because they do not require per-
example gradients. Thus, the optimizations allow training much larger models
without running out of memory. The maximum physical batch size using Private
Vision library is the one that comes closest to the non-private baseline. In
general, we can see that the methods are consistent within implementations,
with Private Vision and the FastDP reaching the same maximum physical batch
size no matter the clipping mode. As expected, the A100 achieves consistently
higher maximum physical batch sizes than the V100 due to the larger amount of
VRAM.

Figure 5: Throughput using the maximum batch size for each clipping algorithm.
It compares the executions for both V100 and A100, for the ViT Base model.

Throughput Comparision Figure 5 displays the throughput for each clipping
algorithm for each available GPU. To understand it better, we also compute the
relative difference between the mean throughput. By using better hardware
components, in this case, improving from a V100 to an A100 GPU has an
increment of $\times 1.3$ times in throughput for all clipping methods. The
one that benefits the most is the per-example clipping by Opacus with a
$\times 1.46$ improvement in throughput. It is because of Opacus-specific
optimizations. Their implementation is optimized to vectorize the virtual
batches and get the most out of the processing unit (Yousefpour et al., 2021)
to compensate for the per-example clipping. Also, we base our virtual batching
module on Opacus, being specially optimized for the Opacus implementation,
yielding better results with better hardware. The other implementations will
have benefits similar to those of non-private training. The clipping methods
in both GPU architectures will have the same relative throughput regarding the
non-private performance. Private Vision gets closer to the non-private
baseline physical batch size, but Book Keeping is closer to its throughput
with a smaller physical batch size (see Figure 6).

Without sacrificing utility, these optimizations offer an alternative to the
original per-example clipping algorithm. Even though Book Keeping performs
better, it is by a very narrow margin. Consequently, Private Vision and FastDP
remain viable options for implementing ghost clipping. The difference between
the two algorithms is the second backward pass over the neural network. Since
the Book Keeping trick avoids doing the second backward pass through the
network, it has a higher throughput at a small memory cost.

Mixing ghost clipping does not yield any improvement because it determines
whether it should apply ghost or per-example clipping, which depends on the
size of the inputs and the parameter space. If the dimensions are large
enough, the ghost technique will be more expensive (Bu et al., 2022). In a ViT
model, the dimensions change less than in a convolutional network. Therefore,
despite continually evaluating which method to apply, it always uses ghost
clipping. However, if applied to a ResNet model, it should outperform ghost
clipping, as it is optimized for convolutional layers. It could not be tested
on BiT ResNet models used in this study due to incompatibilities with the
Private Vision and FastDP, preventing an assessment of mixed optimization
methods.

###  5.2 JAX framework

In this section we compare the performance of JAX with all other DP-SGD
frameworks (all of them are based on PyTorch). To make a fair comparison
between frameworks, we implemented Poisson subsampling and virtual batching,
based on the Opacus implementation.

Compilation time Comparing JAX to PyTorch requires taking the compilation time
into account that the DP-SGD implementations in PyTorch do not utilize. There
is no straightforward way of calculating the compilation time, but we measure
it as the duration to process the first physical batch. The execution times
for each batch shows that the first one takes much more time than the others,
therefore including the compilation time (Figure A.1). To provide a fair
comparison, we also implemented a non-private JAX training using the same
virtual batching as PyTorch.

Throughput comparision In Figure 6, both JAX implementations start at the same
point, since by using virtual batching, they are processing just one sample at
a time and getting the gradients from the compiled function. The non-private
benefits more from a larger batch size, although its throughput decreases at
128 instances. The compilation time (see Figure A.1) grows with the batch
size. For the private model, it takes more time since the compiled function is
more complex than the non-private counterpart. It includes expanding the
dimensions and clipping the gradients, while the non-private directly computes
the gradient of the whole mini-batch.

Figure 6: Comparison of the throughput as a function of the physical batch
size between the JAX and PyTorch clipping algorithms on A100 GPU. Only the
ghost implementations from Private Vision and Book Keeping are used, not the
Mix algorithms, since they have the same performance. The estimator is the
median, and the error bars are the $95\%$ confidence interval using
bootstrapping. All methods use the same estimator and confidence interval, and
JAX is the only one in which the throughput spreads significantly across runs.

JAX per-example clipping reaches a higher physical batch size, and the
throughput is always higher than its Opacus counterpart. However, the Book
Keeping Ghost Clipping implementation is closer to the JAX private version.
They are the same until the physical batch size of 32, but even then, the
variance in the JAX execution is high, meaning that it can even be below the
BK-ghost throughput. JAX private implementation throughput drops when the
batch size doubles to 64. This comparison would be a reason to use Book
Keeping ghost clipping for Vision Transformer models over JAX implementation.
Its throughput is comparable to the execution of compiled code from JAX while
reaching a larger batch size.

JAX private model starts with a similar throughput as the non-private model
and then steadily grows until it surpasses the Torch throughput after a
physical batch size of 16. After that, JAX will always have better throughput,
except at a physical batch size of 128, where JAX throughput drops and is the
same as the PyTorch non-private. We ran the experiment multiple times, and at
the 128 size, it has a low variance, with always the same throughput value.
Another difference between the two frameworks is the variability in the
experiments. PyTorch runs are remarkably consistent in having a low variance,
and the same throughput result is expected every time for a fixed seed.

JAX executions are more variable than those of PyTorch, likely due to its
sensitivity to HPC environment fluctuations and accelerator stochasticity, as
noted in Figure A.1. Another contributing factor is JAX’s asynchronous
dispatch method, which complicates time benchmarking by issuing a promise
rather than immediate results, concealing Python overheads.

Poisson sampling Using JAX for DP introduces complexities, particularly around
subsampling which is crucial for privacy accounting. Implementing Poisson
subsampling results in variable batch sizes in JAX; changes in batch size
require JIT to recompile, leading to graph retracing which is costly and
contributes to execution run variability, as discussed by Chua et al. (2024).

Comparison with PyTorch Although compiling PyTorch is possible, we could not
see any improvements in terms of speed-up. While compiling the non-private
model worked, the speed-up gained was minimal and, in the end, even lower if
we consider the compilation time. PyTorch also recompiles after a batch size
change. While trying to compile, PyTorch falls back to predefined CUDA
operations that are already optimized. In the case of the private setting, the
compilation does not recognize Opacus hooks and continues the execution
without compiling them.

Leveraging the same kernels to support the private hooks and avoid the
compilation would require massive engineering work of writing special kernels
for each specific private case. On the other hand, JAX will compile the JIT
functions in XLA, but it does not fall back to the kernels, making it more
generalizable (Subramani et al., 2021).

JAX reaches a higher accuracy in Table A4 than its PyTorch counterpart. We are
still evaluating what is causing the significant change, even when using the
same hyperparameters and training for just two epochs. The difference is more
prominent for the non-private case, but we still see the DP utility cost.

###  5.3 Lower precision

We consider using lower precision to speed up computational times. We evaluate
the use of Tensor Core 32 (TF32) for training. It has 10 bits for precision,
with eight range bits, the same as 32 single precision (FP32)Kharya (2020).
Using lower precision can have benefits exactly where DP training struggles:
it requires less memory, uses fewer bits to represent the data, and its
operations are optimized for GPU, making them much faster NVIDIA (2023). It is
specially optimized for the A100 GPU and unavailable for the V100, so the
comparison will be between training on the A100 with and without TF32.

Figure 7: Relative difference in mean throughput between TF32 and FP32
Training for ViT Models.

Experimental results In Figure 7, the relative difference between mean
throughput between runs with TF32 and FP32. For non-private training,
throughput increases with model size, peaking at nearly four times the speed
for the ViT Huge model. Then, the private training throughput increases for
the smaller models, but it goes down again as the model size grows after the
base size. The ViT Base model presents the optimal speed-up, suggesting that
the best throughput can be obtained using a middle-size model. One that is too
small does not gain much from TF32, and the larger ones are too expensive. If
we compare the non-private, the growth between the large and huge model sizes
is minimal. The model size is the bottleneck, and the private one is the most
affected by it due to the nature of the clipping algorithm. However, this
remains open for further investigation on why the private training goes up and
down with the model size in lower precision. Regarding the memory advantages
by TF32, we could not see an improvement. Both models, with and without TF32,
could fit the same number of instances.

Concerns regarding TF-32 under DP There are two concerns with using lower
precision in DP deep learning: its effects on utility and privacy. For the
first issue, using lower precision may affect utility, as it is less precise.
We did not find a significant decay in the accuracy of the models; it differs
by decimal points at the $\times 10^{-6}$ precision (See Table A4 in the
Appendix). For example, private training using the optimal number of training
steps, both models reach the same accuracy of $0.879$ with a significant $\sim
2.3$ speed-up thanks to lower precision.

The second concern about privacy was first highlighted by Mironov (2012), but
no fully satisfactory solution exists yet. The problem is that theoretical
guarantees for most algorithms assume computations are performed on reals, and
finite precision arithmetic can lead to actual violations. Discrete mechanisms
(e.g. Canonne et al., 2020; Agarwal et al., 2021) that avoid the theoretical
challenges exist, but they are often less convenient and may lead to loss of
utility, especially in low precision settings. The efficiency of different
discrete mechanisms in TF-32 is an interesting topic of further research.

##  6 Multi-GPU Training

This subsection will look at another angle to train deep learning with DP
faster: increasing the computational resources enough to decrease the training
time. This is relevant when training cost or resource constraints are less
important than the time to train a new model.

###  6.1 Experimental Setup

We utilize V100 GPUs on HPC nodes that have 4 GPUs per node. The other
experimental setting is idential to the one in Section 4. Results for
utilizing up to 24 A100 GPUs can be found in the Appendix Figure A.3.

We focus on comparing the scaling behaivour between the non-private baseline
that uses PyTorch and the DP-SGD implementation using Opacus. Both frameworks
provide mature tooling for distributed training. Although the other clipping
methods do not break, we are still determining if they are correctly handling
the distributed gradient computation and clipping or if distributed training
is even supported.

Figure 8: Comparison between the throughput by scaling the number of GPUs with
more nodes for the non-private and Opacus training with the ViT base model on
V100 GPUs. The dashed line is the ideal growth if it were linear.

###  6.2 Experimental Results

Figure 8 shows the throughput increase as we add more computational resources,
going from one to 80 GPUs. The plot shows that the throughput does not grow
linearly and starts changing from the ideal after using more than one node
(i.e. when using more than 4 GPUs). The communication inside the node is fast,
but the communication between nodes will always be slower. The bottleneck is
the bandwidth, and it prevents the model from scaling linearly. Notably, it
affects the non-private training baseline much more, while the private scales
better. For the 80 GPUs, the private training achieves $69.2\%$ of the ideal
throughput, and the non-private training only achieves $53.3\%$. Private
training scales better because it is slower and only sometimes saturates the
network with updates.

If we use Amdalh’s law to compare the parallelism percentage for each case, we
can see that in the private case, we achieve a $99.5\%$ parallelism compared
to a $98.9\%$ in the non-private case (See Figure A.4 in the Appendix).

##  7 Conclusion

While DP-SGD is significantly more costly than non-private training, we
identified feasible speed-ups that are often easy to apply but have some
drawbacks. These are:  1 More efficient implementations of DP-SGD which
additionally decrease the memory footprint (allowing for training larger
models). However, the implementations are not as mature as Opacus and do not
support all types of neural network layers (yet).  2 JAX which processes
samples faster than PyTorch but looses the advantage through frequent re-
compilations when utilizing proper Poisson sampling, does not offer a
comprehensive DP-SGD implementation as PyTorch and runs not as stable as
PyTorch.  3 Lower Precision using TF-32 which increases the throughput but the
implications on the theoretical guarantees of DP-SGD need to be explored in
future work.  Finally, we found that distributed computing using DP-SGD scales
better than non-private training and allows for fast training of models.

## Acknowledgements

This work was supported by the Finnish Ministry of Education and Culture and
CSC - IT Centre for Science (Decision diary number OKM/10/524/2022), the
Research Council of Finland (Flagship programme: Finnish Center for Artificial
Intelligence, FCAI, Grant 356499 and Grant 359111), the Strategic Research
Council at the Research Council of Finland (Grant 358247) as well as the
European Union (Project 101070617). Niki Loppi contributed under the NVIDIA AI
Technology Center (NVAITC) Finland program. Views and opinions expressed are
however those of the author(s) only and do not necessarily reflect those of
the European Union or the European Commission. Neither the European Union nor
the granting authority can be held responsible for them. This work has been
performed using resources provided by the CSC – IT Center for Science,
Finland. We thank Joonas Jälkö for helpful discussions regarding implementing
DP-SGD with JAX.

## References

  * Abadi et al. (2016) Abadi, M., Chu, A., Goodfellow, I. J., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L.  Deep learning with differential privacy.  In Weippl, E. R., Katzenbeisser, S., Kruegel, C., Myers, A. C., and Halevi, S. (eds.), _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016_ , pp. 308–318. ACM, 2016.  doi: 10.1145/2976749.2978318.  URL https://doi.org/10.1145/2976749.2978318.
  * Abowd (2018) Abowd, J. M.  The U.S. Census Bureau adopts differential privacy.  In Guo, Y. and Farooq, F. (eds.), _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018_, pp. 2867. ACM, 2018.  doi: 10.1145/3219819.3226070.  URL https://doi.org/10.1145/3219819.3226070.
  * Agarwal et al. (2021) Agarwal, N., Kairouz, P., and Liu, Z.  The skellam mechanism for differentially private federated learning.  In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , pp. 5052–5064, 2021.  URL https://proceedings.neurips.cc/paper/2021/hash/285baacbdf8fda1de94b19282acd23e2-Abstract.html.
  * Ansel et al. (2024) Ansel, J., Yang, E. Z., He, H., Gimelshein, N., Jain, A., Voznesensky, M., Bao, B., Bell, P., Berard, D., Burovski, E., Chauhan, G., Chourdia, A., Constable, W., Desmaison, A., DeVito, Z., Ellison, E., Feng, W., Gong, J., Gschwind, M., Hirsh, B., Huang, S., Kalambarkar, K., Kirsch, L., Lazos, M., Lezcano, M., Liang, Y., Liang, J., Lu, Y., Luk, C. K., Maher, B., Pan, Y., Puhrsch, C., Reso, M., Saroufim, M., Siraichi, M. Y., Suk, H., Zhang, S., Suo, M., Tillet, P., Zhao, X., Wang, E., Zhou, K., Zou, R., Wang, X., Mathews, A., Wen, W., Chanan, G., Wu, P., and Chintala, S.  PyTorch 2: Faster machine learning through dynamic Python bytecode transformation and graph compilation.  In Gupta, R., Abu-Ghazaleh, N. B., Musuvathi, M., and Tsafrir, D. (eds.), _Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2024, La Jolla, CA, USA, 27 April 2024- 1 May 2024_ , pp. 929–947. ACM, 2024.  doi: 10.1145/3620665.3640366.  URL https://doi.org/10.1145/3620665.3640366.
  * Balle et al. (2022) Balle, B., Cherubin, G., and Hayes, J.  Reconstructing training data with informed adversaries.  In _43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022_ , pp. 1138–1156. IEEE, 2022.  doi: 10.1109/SP46214.2022.9833677.  URL https://doi.org/10.1109/SP46214.2022.9833677.
  * Bradbury et al. (2018) Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q.  JAX: composable transformations of Python+NumPy programs.  http://github.com/google/jax, 2018.
  * Bu et al. (2022) Bu, Z., Mao, J., and Xu, S.  Scalable and efficient training of large convolutional neural networks with differential privacy.  In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_ , 2022.  URL http://papers.nips.cc/paper_files/paper/2022/hash/fa5617c176e76fee83f3f9947fdf9f3f-Abstract-Conference.html.
  * Bu et al. (2023) Bu, Z., Wang, Y., Zha, S., and Karypis, G.  Differentially private optimization on large model at small cost.  In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 3192–3218. PMLR, 2023.  URL https://proceedings.mlr.press/v202/bu23a.html.
  * Canonne et al. (2020) Canonne, C. L., Kamath, G., and Steinke, T.  The discrete gaussian for differential privacy.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/b53b3a3d6ab90ce0268229151c9bde11-Abstract.html.
  * Carlini et al. (2021) Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T. B., Song, D., Erlingsson, Ú., Oprea, A., and Raffel, C.  Extracting training data from large language models.  In Bailey, M. and Greenstadt, R. (eds.), _30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021_ , pp. 2633–2650. USENIX Association, 2021.  URL https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting.
  * Chua et al. (2024) Chua, L., Ghazi, B., Kamath, P., Kumar, R., Manurangsi, P., Sinha, A., and Zhang, C.  How private is DP-SGD?  _ArXiv preprint_ , abs/2403.17673, 2024.  URL https://arxiv.org/abs/2403.17673.
  * Cormode et al. (2018) Cormode, G., Jha, S., Kulkarni, T., Li, N., Srivastava, D., and Wang, T.  Privacy at scale: Local differential privacy in practice.  In Das, G., Jermaine, C. M., and Bernstein, P. A. (eds.), _Proceedings of the 2018 International Conference on Management of Data, SIGMOD Conference 2018, Houston, TX, USA, June 10-15, 2018_ , pp. 1655–1658. ACM, 2018.  doi: 10.1145/3183713.3197390.  URL https://doi.org/10.1145/3183713.3197390.
  * De et al. (2022) De, S., Berrada, L., Hayes, J., Smith, S. L., and Balle, B.  Unlocking high-accuracy differentially private image classification through scale.  _ArXiv preprint_ , abs/2204.13650, 2022.  URL https://arxiv.org/abs/2204.13650.
  * DeepMind et al. (2020) DeepMind, Babuschkin, I., Baumli, K., Bell, A., Bhupatiraju, S., Bruce, J., Buchlovsky, P., Budden, D., Cai, T., Clark, A., Danihelka, I., Dedieu, A., Fantacci, C., Godwin, J., Jones, C., Hemsley, R., Hennigan, T., Hessel, M., Hou, S., Kapturowski, S., Keck, T., Kemaev, I., King, M., Kunesch, M., Martens, L., Merzic, H., Mikulik, V., Norman, T., Papamakarios, G., Quan, J., Ring, R., Ruiz, F., Sanchez, A., Sartran, L., Schneider, R., Sezener, E., Spencer, S., Srinivasan, S., Stanojević, M., Stokowiec, W., Wang, L., Zhou, G., and Viola, F.  The DeepMind JAX Ecosystem.  http://github.com/google-deepmind, 2020.
  * Dosovitskiy et al. (2021) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.  URL https://openreview.net/forum?id=YicbFdNTTy.
  * Dwork et al. (2006) Dwork, C., McSherry, F., Nissim, K., and Smith, A. D.  Calibrating noise to sensitivity in private data analysis.  In Halevi, S. and Rabin, T. (eds.), _Theory of Cryptography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006, Proceedings_ , volume 3876 of _Lecture Notes in Computer Science_ , pp. 265–284. Springer, 2006.  doi: 10.1007/11681878˙14.  URL https://doi.org/10.1007/11681878_14.
  * Hu et al. (2022) Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W.  Lora: Low-rank adaptation of large language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.  URL https://openreview.net/forum?id=nZeVKeeFYf9.
  * Kharya (2020) Kharya, P.  TensorFloat-32 in the A100 GPU Accelerates AI Training, HPC up to 20x.  https://blogs.nvidia.com/blog/tensorfloat-32-precision-format/, 2020.
  * Kolesnikov et al. (2020) Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N.  Big transfer (BiT): General visual representation learning.  In _Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V_ , pp. 491–507, Berlin, Heidelberg, 2020. Springer-Verlag.  ISBN 978-3-030-58557-0.  doi: 10.1007/978-3-030-58558-7˙29.  URL https://doi.org/10.1007/978-3-030-58558-7_29.
  * Krizhevsky & Hinton (2009) Krizhevsky, A. and Hinton, G.  Learning multiple layers of features from tiny images.  Technical Report 0, University of Toronto, Toronto, Ontario, 2009.  URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.
  * Li et al. (2022) Li, X., Tramèr, F., Liang, P., and Hashimoto, T.  Large language models can be strong differentially private learners.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.  URL https://openreview.net/forum?id=bVuP3ltATMz.
  * Mironov (2012) Mironov, I.  On significance of the least significant bits for differential privacy.  In Yu, T., Danezis, G., and Gligor, V. D. (eds.), _Proceedings of the 2012 ACM Conference on Computer and Communications Security_ , CCS ’12, pp. 650–661, New York, NY, USA, 2012. Association for Computing Machinery.  ISBN 9781450316514.  doi: 10.1145/2382196.2382264.  URL https://doi.org/10.1145/2382196.2382264.
  * NVIDIA (2023) NVIDIA.  Train with mixed precision.  https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html, 2023.
  * Perez et al. (2018) Perez, E., Strub, F., de Vries, H., Dumoulin, V., and Courville, A. C.  Film: Visual reasoning with a general conditioning layer.  In McIlraith, S. A. and Weinberger, K. Q. (eds.), _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pp. 3942–3951. AAAI Press, 2018.  URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16528.
  * Ponomareva et al. (2023) Ponomareva, N., Vassilvitskii, S., Xu, Z., McMahan, B., Kurakin, A., and Zhang, C.  How to dp-fy ML: A practical tutorial to machine learning with differential privacy.  In Singh, A. K., Sun, Y., Akoglu, L., Gunopulos, D., Yan, X., Kumar, R., Ozcan, F., and Ye, J. (eds.), _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023_ , pp. 5823–5824. ACM, 2023.  doi: 10.1145/3580305.3599561.  URL https://doi.org/10.1145/3580305.3599561.
  * Räisä et al. (2024) Räisä, O., Jälkö, J., and Honkela, A.  Subsampling is not magic: Why large batch sizes work for differentially private stochastic optimisation.  _ArXiv preprint_ , abs/2402.03990, 2024.  URL https://arxiv.org/abs/2402.03990.
  * Rajkumar & Agarwal (2012) Rajkumar, A. and Agarwal, S.  A differentially private stochastic gradient descent algorithm for multiparty classification.  In Lawrence, N. D. and Girolami, M. A. (eds.), _Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2012, La Palma, Canary Islands, Spain, April 21-23, 2012_ , volume 22 of _JMLR Proceedings_ , pp. 933–941. JMLR.org, 2012.  URL http://proceedings.mlr.press/v22/rajkumar12.html.
  * Song et al. (2013) Song, S., Chaudhuri, K., and Sarwate, A. D.  Stochastic gradient descent with differentially private updates.  In _IEEE Global Conference on Signal and Information Processing, GlobalSIP 2013, Austin, TX, USA, December 3-5, 2013_ , pp. 245–248. IEEE, 2013.  doi: 10.1109/GlobalSIP.2013.6736861.  URL https://doi.org/10.1109/GlobalSIP.2013.6736861.
  * Steiner et al. (2022) Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., and Beyer, L.  How to train your ViT? Data, augmentation, and regularization in vision transformers.  _Transactions on Machine Learning Research_ , 2022, 2022.  URL https://openreview.net/forum?id=4nPswr1KcP.
  * Subramani et al. (2021) Subramani, P., Vadivelu, N., and Kamath, G.  Enabling fast differentially private SGD via just-in-time compilation and vectorization.  In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , pp. 26409–26421, 2021.  URL https://proceedings.neurips.cc/paper/2021/hash/ddf9029977a61241841edeae15e9b53f-Abstract.html.
  * Tobaben et al. (2023) Tobaben, M., Shysheya, A., Bronskill, J., Paverd, A., Tople, S., Béguelin, S. Z., Turner, R. E., and Honkela, A.  On the efficacy of differentially private few-shot image classification.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=hFsr59Imzm.
  * Wightman (2019) Wightman, R.  PyTorch image models.  https://github.com/huggingface/pytorch-image-models, 2019.
  * Yousefpour et al. (2021) Yousefpour, A., Shilov, I., Sablayrolles, A., Testuggine, D., Prasad, K., Malek, M., Nguyen, J., Ghosh, S., Bharadwaj, A., Zhao, J., Cormode, G., and Mironov, I.  Opacus: User-friendly differential privacy library in PyTorch.  _ArXiv preprint_ , abs/2109.12298, 2021.  URL https://arxiv.org/abs/2109.12298.
  * Yu et al. (2022) Yu, D., Naik, S., Backurs, A., Gopi, S., Inan, H. A., Kamath, G., Kulkarni, J., Lee, Y. T., Manoel, A., Wutschitz, L., Yekhanin, S., and Zhang, H.  Differentially private fine-tuning of language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.  URL https://openreview.net/forum?id=Q42f0dfjECO.

##  Appendix A Training Details

###  A.1 Framework and implementation compatibility

Table A1 shows the corresponding support between implementations and clipping
methods. We also include general frameworks, like PyTorch and JAX. Opacus,
PrivateVision, and FastDP are PyTorch based implementations.

Table A1: Clipping optimization and the library or framework that implements
it.
Clipping Mode | PyTorch | Opacus | PrivateVision | FastDP | JAX
---|---|---|---|---|---
Non-private | $\surd$ |  |  |  | $\surd$
Per-example |  | $\surd$ |  |  | $\surd$
Ghost Clipping |  |  | $\surd$ | $\surd$ |
Mix Ghost |  |  | $\surd$ | $\surd$ |
Mix Opt |  |  |  | $\surd$ |

###  A.2 Models

  * •

Vision Transformer (ViT) Dosovitskiy et al. (2021). Taken from
https://huggingface.co/timm/vit_base_patch16_224.augreg2_in21k_ft_in1k

  * •

Big Transfer BiT-ResNet Kolesnikov et al. (2020). Taken from
https://github.com/google-research/big_transfer

###  A.3 Hyperparameters

We use the hyperparameters obtained on request from Tobaben et al. (2023). The
hyperparameters for both models are in Table A2. We do not optimize them
futher, as model utility is not the main objective in this work.

Table A2: Hyperparameters used for each model architecture.
Model | Trainable Parameters | Epsilon | Delta | Learning Rate | Max Grad Norm
---|---|---|---|---|---
ViT | All | 8 | $2.04e^{-5}$ | 0.0003 | 4.63
BiT-ResNet | All | 8 | $2.04e^{-5}$ | 0.00098 | 6.53

##  Appendix B Additional Results

This section provides additional figures that supplement the findings in the
main text.

Figure A.1: Compilation time in seconds as a function of the physical batch
size for JAX experiments for the ViT Base model on A100. The estimator is the
median and the error bars are the $95\%$ confidence interval using
bootstrapping.

Figure A.2: Combining distributed training with the use of lower precision
TF32 for the ViT base model on A100. A.2 Throughput for one GPU; A.2
Throughput for multiple GPUs. Figure A.3: Comparison between the throughput by
scaling the number of GPUs with more nodes for the non-private and Opacus
training with the ViT base model on A100 GPUs. The dashed line is the ideal
growth if it were linear.

Figure A.4: Comparison between the throughput in our experiments and the
theoretical Amdahl’s Law. Both axis are in log scale. In the distributed
setting, private training achieves a 99.5 $\%$ of parallel processing, with a
50 times speed up than single processing.  Table A3: Average processing time
for each section of the algorithm. We are comparing the non-private and Opacus
per-example clipping on A100, with the same physical batch size. It is
calculated with NVIDIA Nsight Systems. All the measurements include the
syncronization time, which is needed for the profiling, but adds additional
time that is not part of the normal execution. All values are in milliseconds.
Section | PyTorch non-private | Opacus per-example
---|---|---
Forward | 81.14 | 101.53
Backward | 163.85 | 681.48
Clip and Accumulation | 0 | 26.76
Optimizer Step | 38.17 | 99.65
Table A4: Mean accuracy for CIFAR-100 test set for each clipping mode for the
ViT models on A100 after training for two epochs. All use the ViT
hyperparameters from Table A2. While this work does not focus on the model’s
utility, having their results still allows us to compare them. Using optimal
hyperparameters for DP causes low utility in non-private training. The use of
TF32 as a lower precision mode does not affect the model’s utility. We are
still analyzing why JAX accuracy is significantly higher in both cases, even
when it uses sub-optimal hyperparameters.
Clipping Mode | Test Accuracy
---|---
Non-private | 0.2285
Non-private/TF32 | 0.2301
Non-private JAX | 0.8273
O-flat Opacus | 0.7096
O-flat Opacus/TF32 | 0.7094
O-flat JAX | 0.8009
PV-ghost | 0.6978
BK-ghost | 0.7341


# Analog Seizure Detection for Implanted Responsive Neurostimulation

Abbas A. Zaki1, Noah C. Parker1, Tae-Yoon Kim1, Sam Ishak1, Ty E. Stovall1,
Genchang Peng1,
Hina Dave2, Jay Harvey2, Mehrdad Nourani1, Xuan Hu1, Alexander J. Edwards1,
Joseph S. Friedman1  1Department of Electrical and Computer Engineering, The
University of Texas at Dallas, Richardson, TX
{abbas.zaki, noah.parker1, taeyoon.kim, sam.ishak, ty.stovall, gxp170004,
nourani, xuan.hu, alexander.edwards<EMAIL_ADDRESS>2Department of Neurology, The University of Texas Southwestern Medical Center,
Dallas, TX
{hina.dave<EMAIL_ADDRESS>
###### Abstract

Epilepsy can be treated with medication, however, 30% of epileptic patients
are still drug resistive. Devices like responsive neurostimluation systems are
implanted in select patients who may not be amenable to surgical resection.
However, state-of-the-art devices suffer from low accuracy and high
sensitivity. We propose a novel patient-specific seizure detection system
based on naïve Bayesian inference using Müller C-elements. The system improves
upon the current leading neurostimulation device, NeuroPace’s RNS® by
implementing analog signal processing for feature extraction, minimizing the
power consumption compared to the digital counterpart.

Preliminary simulations were performed in MATLAB, demonstrating that through
integrating multiple channels and features, up to $98\%$ detection accuracy
for individual patients can be achieved. Similarly, power calculations were
performed, demonstrating that the system uses $6.5\mu W$ per channel, which
when compared to the state-of-the-art NeuroPace system would increase battery
life by up to $50\%$.

###### Keywords:

Seizure Detection; Refractory Epilepsy; Responsive Neurostimulation; Analog
Signal Processing; Stochastic Computing; Naïve Bayesian Inference

##  I Introduction

As one of the most common neurological conditions, epilepsy affects 50 million
people worldwide [1]. Epilepsy is characterized by recurrent seizures
resulting from excessive electrical discharges in either focal or generalized
regions of the brain [1]. Seizure types can vary from brief lapses of
attention to muscle jerks and debilitating convulsions [1]. Approximately 30%
of epileptic patients suffer from refractory or drug-resistant epilepsy [2].
Surgical resection is the conventional approach to treating drug-resistant
focal epilepsy. However, resective surgery is not an option for all patients
suffering from drug-resistant epilepsy especially if the seizures lack a well-
defined epileptogenic region or if the region is in the eloquent cortex/deep
brain structures. Furthermore, some patients continue to have seizures even
after surgical resection [3].

As surgical resection is not always possible or helpful, implantable
neuromodulatoy devices delivering electrical stimulations to targeted regions
of the brain are the next best choice to reduce or mitigate seizures [4, 5,
2]. Two intracranial options include deep brain stimulation (DBS) and
responsive neurostimulation. The currently FDA-approved DBS device is an open-
loop system acting similar to a pace-maker offering regular pulses with no
feedback from sensors in the brain. In contrast, responsive neurostimulation
devices measure neural activity and offer stimulation when seizure activity is
detected. Historically there has been a two-fold trade off between detecting
as many seizures as possible and not overstimulating the patient without
compromising battery life. In the extreme cases, embedded deep leaning models
for seizure detection have been proposed which would be very power hungry,
and, on the other extreme, some systems have limited to no seizure detection
relying on over-stimulation to catch more seizure activity.

We propose here a solution to this problem taking advantage of the fact that
seizure activity is often distributed in various networks throughout the
brain. We propose a solution using multiple low-power inferences from several
EEG channels, combining potentially disagreeing evidences with an extremely
low-power Bayesian inference engine comprising C-elements and stochastic bit-
streams [6]. Preliminary simulations are performed demonstrating high
detection accuracy, and a high-level power calculation is performed
demonstrating that our detection engine can increase battery life by up to 50%
as compared with the state-of-the-art RNS® device developed by NeuroPace.

##  II Background

###  II-A Previous Seizure Detection Solutions

Seizure detection and treatment have been of great interest in recent years.
Implantable devices are of special interest in the clinical world because they
can greatly improve the quality of life for patients dealing with chronic
clinical seizures. Most notably, NeuroPace’s RNS® system stands out for being
the only FDA-approved responsive neurostimulation system on the market.
NeuroPace’s system comprises a battery, controller, and one to two probes for
EEG data acquisition and neurostimulation, in which charge controllably passes
through brain tissue between selected contacts on the probes.

For being the first system to market in this field, Neuropace’s RNS® system is
outstanding—but it has its limitations. Notably, its method for seizure
detection is not stellar, evaluating individual low-precision features to
detect seizures. This is intentional to minimize power usage. However, one
major limitation of this is that as the detection scheme is not highly
accurate, clinicians end up tuning these systems over months to years to be
over-sensitive and to overstimulate the brain. NeuroPace claims that during
clinical studies, the device would stimulate patients roughly 600 times per
day for less seizure-active patients and up to 2000 times a day for patients
with increased seizure activity. Though there are no known clinical drawbacks
to over-stimulation, it is wasteful from a power management perspective, and
based on a rough calculation from numbers in [7] stimulation uses up to 20% of
the battery life depending on the patient’s neural activity and the
stimulation parameters. With a better inference engine, hyper-sensitivity
could be reduced and battery life could be significantly extended while
retaining the demonstrated benefits of responsive neurostimulation systems.

For maximizing the battery life of an implantable seizure treatment device,
then, the difficulty lies in balancing the energy usage from high-accuracy
feature evaluation and the energy lost to over-stimulation. There are a number
of resources demonstrating accuracy and energy for different types of
features, and the results are pretty uniform: looking at single channel,
exotic features (often implemented with Digital Signal Processing (DSP) blocks
or neural networks) can achieve stellar accuracies but are very power hungry
[8, 9]. Simultaneously, low-power features can have significantly lower
accuracy [10, 11], though when multiple low-power features are analyzed
jointly, that accuracy can be improved [10, 11]. Thus, a low power solution
can be found in integrating evidences from multiple channels evaluated with
low-power features. The question remaining is whether there is a low-power
solution to integrate these potentially conflicting, low-power evidences from
multiple channels simultaneously. There is.

###  II-B Bayesian Inference with Müller C Elements

We previously proposed an extremely low-power implementation of Bayesian
inference using C-Elements (C-Es) and stochastic bitstreams [6]. A diagramatic
representation of the Müller C-element and its Truth table can be seen in Fig.
1. In brief, Bayesian inference computes the probability of an event $V$ given
the $i$ observed evidences $E^{m}_{1},E^{m}_{2},...E^{m}_{i}$ where $m$ is the
observed value of the corresponding evidence, so $P(E^{0.5}_{3})$ corresponds
to the probability that evidence 3 has the value of $0.5$. With this notation,
naïve Bayesian inference is formulated as:

$P(V\mid{E^{m}_{1},...,E^{m}_{i}})\\\
=\frac{P(V)\prod^{i}_{x=1}P^{*}(E^{m}_{x})}{P(V)\prod^{i}_{x=1}P^{*}(E^{m}_{x})+(P(\bar{V}))\prod^{i}_{x=1}(1-P^{*}(E^{m}_{x}))},$
(1)

where $P^{*}(E^{m}_{x})$ is defined as

$P^{*}(E^{m}_{x})=\frac{P(E^{m}_{x}\mid V)}{P(E^{m}_{x}\mid
V)+P(E^{m}_{x}\mid\bar{V})}.$ (2)

Stochastic computing is a recently proposed methodology for approximate
computing [12]. Real numbers in $[0,1]$ are encoded as the probability that at
any given time the state of a stochastic bitstream will be $1$. Stochastic
computing offers new ways of computing mathematical operations; for instance,
an AND gate can perform multiplication and a MUX can be used for weighted
addition [12]. Most notably [6] recognized that multi-inut a Müller C-E block
in the stochastic regime can compute the function:

$Q=\frac{\prod^{i}_{x=1}D_{x}}{\prod^{i}_{x=1}D_{x}+\prod^{i}_{x=1}(1-D_{x})}$

Which when the $D_{x}$ values are interpreted as $P(V)$ and
$P^{*}(E^{m}_{x})$, directly implements eq. (1). Thus, in the stochastic
regime a single multi-input C-E [13, 6] will directly implement Bayesian
inference. In [6] analysis was performed demonstrating that with minor
accuracy loss, energy usage can be heavily reduced to the order of $nJ$ per-
inference.

Ultimately then, from a high-level view of the field, for high-accuracy and
low-power seizure detection, multiple low-power features across different
channels should be analyzed and integrated using low-power inference. We
propose such a system which does not require high-power analog-to-digital
conversion (ADC) and digital signal processing (DSP) blocks, by instead making
use of alternative analog and stochastic blocks for energy-efficient, high-
accuracy seizure detection.

Figure 1: a) Multi-input C-Element for Bayesian inference. b) C-Element truth
table. c) C-Element circuit implementation. Figure 2: Data path from EEG
signal acquisition to seizure detection.

##  III System Overview

To enable an extremely low-power solution for seizure detection avoiding DSP
computations and ADCs, analog processing blocks are used as much as possible.
Fig. 2 depicts the complete data path from EEG signal extraction to seizure
identification.

To minimize power usage, the system will not store or transmit EEG data;
rather it’s sole purpose will be to detect and treat seizures. This heavily
limits the adjustability of the system after implantation. In this work, it is
assumed that before surgery, copious information about the patient’s seizure
networks has been gathered, and proper EEG channels and associated features
can be identified and tuned before implantation surgery. From our analysis in
section IV the number of $(feature,channel)$ pairs that need to be extracted
for high-accuracy detection varies from one to four depending on the patient.

This pre-surgery study of the patient will govern the placement of probes in
the brain. EEG voltage signals range from hundreds of micro-volts to tens of
milli-volts and information is encoded in frequency bands of tens of hertz to
hundreds of hertz. These somewhat low-frequency and low-voltage signals can be
amplified by a low-noise differential amplifier to produce an analyzable rail-
to-rail signal. One amplifier will be needed per channel analyzed.

Once these signals are obtained, they may be fed into one of several types of
features including mean, energy, mean of absolute value, line lenght, Hjorth
mobility, etc. each of which has a low-power approximate analog implementation
to avoid the high power cost of DSP. Many of these features require
integrating information over a window of time. In our initial discrete
simulations, it was found that a window of roughly five seconds contained
enough information to identify a seizure. Thus, for the analog implementations
of these features, time constants for decaying integration were chosen on the
order of five seconds.

Emerging from each feature will be a level voltage $v_{f}(t)$ corresponding to
the feature value at time $t$. This voltage must be prepared for Bayesian
inference by being transformed to represent $P^{*}(E^{v}_{f})(t)$ introduced
in eq. 2. During the pre-surgery study period, the function
$p:v_{f}\rightarrow P^{*}(E^{v}_{f})$ will be defined for each useful feature
for each channel indicating how likely a seizure is, given the observed
feature value $v_{f}(t)$. The function $p$ will be hardcoded into a voltage-
in, voltage-out lookup table (LUT) which can be implemented through multiple
means. Ideally, it can be closely matched with a low-power analog
implementation. Likewise, we demonstrate below in Sec. IV-A that only a small
number of bins per feature need to be created for high-accuracy detection, so
a series of low-power level detectors can be substituted without loss of
accuracy. As stated above, the system will not be adjustable post-
implantation, so an analog lookup table may be possible here where it is not
in other applications.

Once the voltages representing $P^{*}(E^{v}_{f})(t)$ are obtained, they are
directly fed into a voltage-controlled tunable random number generator (RNG)
which produces stochastic bitstreams with a time-varying probability equal to
$P^{*}(E^{v}_{f})(t)$. Multiple low-footprint tunable RNGs have been recently
shown some utilizing super-paramagnetic magnetic tunnel junctions (MTJs) to
provide entropy [14, 15, 16, 17, 18, 19].

These bitstreams are fed asynchronously into the C-E(s) to perform naïve
Bayesian inference as described in [6]. The output of this network will be a
final stochastic bitstream representing the probability that the patient is
currently in an ictal (seizure) state. Passing this bitstream through a
passive low-pass filter is sufficient to convert this bitstream to a level
voltage. A voltage above a pre-determined threshold will represent a seizure
state. After the threshold is applied, this Boolean signal may be fed into a
treatment subsystem (not shown) to stimulate the seizure network and stifle
the seizure before it causes adverse symptoms.

All of the data-path blocks described above are designed to be as energy
efficient as possible while still providing high-accuracy detection. The
following two sections demonstrate preliminary results showing high inference
accuracy and very low power respectively.

Figure 3: Seizure Detection with Discrete Windows. For each 5-second window,
the system identifies whether there is a seizure or not for both Ictal and
Inter-Ictal states keeping false negatives and false positives to a minimum.
Figure 4: Windowed Single Feature Performance. The highest detection
accuracies for each feature for each patient spanning the 24 EEG channels are
depicted here, demonstrating that simple features can be used to obtain high
detection accuracies for some patients.

##  IV Inference Accuracy

While intracranial EEG data would be ideal for an implantable seizure
detection system, the lack of publicly available databases necessitates the
use of scalp EEG data to demonstrate the system’s viability as a low-power
detection device. It is important to note though that scalp EEG signals
contain more noise, as the electric potentials being measured face
interference from bone and skin; therefore, a system that can reliably detect
seizures with low power on scalp EEG data would be expected to perform equally
well, if not better, on intracranial EEG data albeit with some modifications
to thresholds. As such, the Temple University Seizure Corpus (TUSZ) [20] was
chosen to evaluate the potential effectiveness of the proposed system. The
corpus provides one of most expansive collections of seizure and non-seizure
EEG recordings and contains clinical annotations marking periods of EEG data
that contain seizures, granting a nice interface to demonstrate the high
detection accuracies we were able to achieve with our methodology.

###  IV-A Ideal Windowing and Features

Initial simulations were performed in the digital domain. Patients from the
TUSZ database were selected for study if they had seizures that lasted 60
seconds or longer and had non-seizure activity that lasted 60 seconds. These
patients were evaluated individually to determine the effectiveness of our
proposed system. Each EEG signal was processed using a bipolar Temporal
Central Parasagital (TCP) linked-ears reference montage [20] so that each EEG
recording contained 22 channels of EEG activity.

The seizure and non-seizure activity for each patient was then grouped into
5-second windows and labeled as containing seizure or non-seizure data as
shown in Fig. 3. For each patient, each feature was evaluated on each channel
for each 5-second window. None of the features rely on Fourier or wavelet
transforms as used in other works to avoid incurring high computational costs
associated with each transformation into the frequency/spectral domain.

From here the calculated features were evaluated according to how well they
could be used to detect seizures. For the model, a probability table
corresponding to the probability distribution function for each feature-
channel pair was generated from the training set. This table gives the
probability of observing a specific feature value in an ictal vs. inter-ictal
state, or $P(feature|ictal)$. A non-uniform binning algorithm was used here to
ensure that there was a statistically useful amount of data in each bin and to
limit the overall number of bins.

Figure 5: Analog features. A raw EEG signal along with extracted features is
shown. Simulink blocks used to implement these features are likewise depicted.

Naïve Bayesian inference was then computed directly from this probability
distribution to classify each window in the testing set as showing either
ictal or inter-ictal data. The J-statistic was used to consolidate false-
negatives, false-positives, etc. into a single metric on $[-1,1]$ where a
value of $-1$ implies no classifications were made correctly, and a value of
$1$ corresponds to perfect classification. Results for each patient are shown
in Fig. 4. For each feature, only the J-statistic for the channel with the
highest performance is shown for brevity. High classification accuracies were
obtained for some feature-channel combinations for some patients indicating
the ability for these features to be used to classify focal seizures.

Figure 6: Analog Single Feature Performance. For each feature, only the
highest accuracy obtained among the 24 EEG channels is depicted. Patients
1543, 3401, 5625, and 7623 saw significant increases in accuracy when
integrating information from multiple channels and features. Figure 7: Analog
Multi Feature Performance. Patients 1543, 3401, 5625, and 7623 saw significant
increases in accuracy when integrating information from multiple channels and
features. Figure 8: J Statistic vs Number of Features for selected patients.
The accuracies for the patients highlighted in Figs. 6 and 7 and are tracked
according to how many features were used in each evaluation. Some patients saw
up to a $30\%$ increase in accuracy, and trends indicate that these accuracies
will continue to increase with the further addition of features.

###  IV-B Analog Feature Simulation and Results

To secure validate this result, the same simulations were performed in the
analog domain using Matlab’s Simulink where EEG signals were not discretely
sampled into 5-second windows, but rather, the various time-constants of the
feature circuits emulated a memory window of approximately 5-seconds. Fig. 5
depicts how a number of these features transform the input EEG signals. After
circuit simulation was completed, the feature values were binned and used to
perform naïve Bayesian inference as before. For this experiment, a 5-fold
cross-validation procedure was implemented as in [21, 22, 23, 24] wherein the
data set was partitioned into five subsets. Five trials were run on the data
set, each time using a different subset of the data as the testing set and the
other four subsets as the training set. This was done to increase confidence
in our results obtained from a relatively small data set for each patient.

Fig. 6 displays these results showing agreement with the analysis in section
IV-A. Note that some of the features from Sec. IV-A were not simulated here,
though all of these features have an approximate, low-power analog
implementation. It is expected that the results from the ideal versions of
these features will extend to the analog domain as they do with the features
presented here.

Classification based on a single $(feature,channel)$ pair may be effective for
patients with very focal seizures, but is not effective for many patients
whose seizures are distributed across the brain. Analog domain simulations
were extended to utilize multiple features across multiple channels
simultaneously using naïve Bayesian inference. All combinations of two
features were evaluated to identify the highest performing pair. While
exhaustive evaluation of all possible three and four feature combinations was
not feasible, a genetic algorithm optimizer was seeded with the highest
performing two feature pair and used to find high performing three and four
feature combinations. The results of these optimizer simulations is shown in
Fig. 7.

Our results demonstrate the feasibility of high-accuracy seizure detection
using low-power analog features and Bayesian inference. $93\%$ of patients saw
an increase in seizure classification accuracy using two feature-channel
pairs, and a number of patients continued to see an increase in accuracy
through the case of four feature-channel pairs, several of which are
highlighted in Fig. 8. It is anticipated that combinations of five or more
$(feature,channel)$ pairs can be used to increase these accuracies further.
The best performing patient saw a $30\%$ accuracy increase between one and
four features.

We have demonstrated then that for patients with seizure networks, multiple
features and channels can and should be used to boost detection accuracy,
decreasing false positives and ultimately the number of stimulations issued to
the brain. What remains to be shown here is that these high detection
accuracies can be obtained in a sufficiently low-power context to ensure long
battery life for the highest quality of life for the patient.

##  V Power Analysis

We obtain an upper bound on the power consumption of the detection system. We
perform power analysis of each individual block of Fig. 2 with the system
parameters in Table I. Assuming a battery with $3.3Wh$ capacity as in [7], the
detection system is able to maintain high-accuracy detection using only $p\mu
W$ of power.

TABLE I: System Parameters

Parameter |  Value
---|---
Supply Voltage | 1.2 V
Sampling Frequency | 1000 Hz
Average stimulation current | 1 mA
Maximum stimulation current | 12 mA
Lead Resistance | 1200 ohms
Stimulation Pulse Width | 160 $\mu$s
Stimulation Period | 5 ms
Average Stimulation Duration | 100 ms
Maximum Stimulation Duration | 5000 ms
Parasitic Capacitance | 100 fF
Load Capacitance | 20 pF
Battery Capacity | 3.3 Wh

###  V-A Signal Sensing and Pre-Amplification

The first step of obtaining EEG signals is to detect brain activity through
electrodes. After the electrodes detect the signals, a preliminary low-noise
amplifier is needed to amplify $mV$ or $\mu V$ signals to sub-volt levels for
the cascading circuits to efficiently and accurately process the EEG
signals[25]. As in [26, 27, 28], the current consumption for the preliminary
EEG signal amplification process process is less than $1\ \mu A$ per channel.
As the EEG signals generally have an upper bandwidth limit of $100\ Hz$,
sampling such EEG signals will not introduce notable extra energy with Nyquist
sampling [27]. The estimated power consumption for sensing and pre-
amplification is therefore bound at $1.2\ \mu W$ per $(feature,channel)$.

###  V-B Feature Extraction

To evaluate the power consumption of feature extraction circuits, we implement
four feature evaluation methods as shown in Table II that are used for the
simulations in Section IV. As shown in the block diagram of Fig. 5, the core
analog computing blocks for feature extraction are based on multipliers and
op-amps. The simulations are performed in Cadence with 65 nm technology node
with a 1.2 V supply voltage. Each simulation is run for 10 seconds for the
power calculation with the input signal frequency up to $100\ Hz$, which
matches EEG signal bandwidth. Table II summarizes the simulation results. As
can be seen in Table II, the maximum power consumption of feature extraction
per channel is $3.539\ \mu W$, and the average power consumption of feature
extraction per channel is $2.523\ \mu W$.

TABLE II: Power consumption of EEG feature extraction circuits

Feature name |  Power
---|---
Feature_mean | 1.995 $\mu$W
Feature_mean_abs | 3.539 $\mu$W
Feature_mean_energy | 1.680 $\mu$W
Feature_energy_mean | 2.879 $\mu$W

###  V-C Analog Lookup Table

To convert the results from the feature extraction circuits to a voltage that
modulates the switching probability of the RNGs, an analog-to-analog lookup
table circuit is used. As in Section III this circuit will be different for
each patient for each feature, so there will be various implementations
depending on the function the circuit is representing. In the worst case, a
set of discrete level detectors will be used, and this is the circuit we
analyze here in order to provide an upper bound for the power usage of the
system.

A multilevel voltage reference circuit is used to generate both the
referencing and driving voltages, while a selector network is used to
arbitrarily map the input and output voltages. This circuit is simulated in GF
65nm technology; a eight-level analog lookup table circuit consumes $493.80nW$
of power which scales linearly with the number of discrete input levels. The
architecture of this circuit is preliminary, and it is anticipated that power
consumption can be optimized further. In our simulations in Section IV
approximately 40 discrete bins were used to group feature and probability
data. Using linear scaling then, we estimate a worst case power consumption of
$2.469\mu W$ per $(feature,channel)$ for this block.

###  V-D Stochastic Bitstream Generator

As in [6] and Section II-B, random number generators (RNGs) are required for
the Bayesian inference circuit with C-elements. While conventional CMOS-based
RNGs are also available for low-energy random bitstream generation [16, 15,
14], we propose to use stochastic p-bits [19] which can be used to generate
true random numbers with low energy cost [17] given the low energy barrier of
the nanomagnets [19]. As in [17], the true random number generator (TRNG)
consumes $20\ \text{fJ/bit}$. Assuming a 1000 Hz sample rate which will
produce sufficiently long bitstreams for proper classification, power
consumption per $(feature,channel)$ for the TRNG can be estimated as:

$P_{TRNG}=E_{bit}\cdot f_{s}=20\ \text{pW},$ (3)

where $P_{TRNG}$ is the power consumption for the entire TRNG block of the
proposed system, $E_{bit}$ is the energy required to generate one bit of
random number, and $f_{s}$ is the sample rate.

Compared to other analog blocks within the proposed system, the RNG consumes
significantly less power regardless of whether the RNGs are constructed by
emerging technology devices or conventional CMOS circuits [16, 15, 14].

###  V-E Müller C-Elements

The proposed system requires a multi-input C-element circuit as shown in Fig.
1. As in [6], each two-input C-Element consumes $20fJ$ per bit operation.
Since the power consumed by the C-element blocks are more than three orders of
magnitude less than the major power consumers of the system, the power
consumption of one n-input C-element for n channels can be estimated to be the
same as n two-input for all channels, which can be normalized to one C-element
per $(feature,channel)$. Therefore, the power consumed by each
$(feature,channel)$ of the C-Elements ($P_{C-Element}$) can be estimated as

$P_{C-Element}=1000\ \text{Hz}\cdot 20\ \text{fJ}=20\ \text{pW}.$ (4)

###  V-F Neurostimulation

According to [29, 7], the typical responsive stimulation to seizures is a
series of pulses with a frequency of $200Hz$, individual pulse-widths of
$160\mu s$, $6mA$ amplitude, and total active duration of 100 ms. For a 1200
ohm lead resistance, each stimulation activity consumes:

$E_{stim,avg}=I_{stim}^{2}\cdot R_{lead}\cdot T\cdot D=138\mu J,$ (5)

where $E_{stim,avg}$ is the energy required to generate a series of pulses for
one stimulation activity with average strength; $I_{stim}$ is the stimulating
current amplitude; $R_{lead}$ is the lead resistance; $T$ is the duration of
one stimulation activity; and $D=160\mu s\cdot 200Hz=3.2\%$ is the duty cycle.

When tuned for the patients who need more stimulations, with $I_{s}tim=12mA$,
and a $200\mu s$ pulse width, we get:

$E_{stim,high}=691\mu J,$ (6)

###  V-G Total Power Estimation

The total power consumption per $(feature,channel)$ for the detection system
is:

$P_{total}=P_{sen}+P_{pre-amp}+P_{FE-avg}+P_{LUT}\\\
+P_{TRNG}+P_{C-Element}=6.189\ \mu\text{W}$ (7)

Here, $P_{sen}$, $P_{pre-amp}$, $P_{FE-avg}$, $P_{LUT}$, $P_{TRNG}$, and
$P_{C-Element}$ represent the power consumption on sensing, pre-amplification,
(average) feature extraction, look-up table, true random number generator, and
C-element blocks respectively. Ultimately, our detection system uses about
$6.189\mu W$ of power per $(feature,channel)$ for detection. Preliminary
simulation results indicate that detection accuracies will continue to improve
with an increasing number of $(feature,channel)$ pairs. Assuming that an
average of four $(feature,channel)$ pairs are used per patient, the detection
circuitry consumes roughly $25\mu W$. For stimulation, average seizure-
activity patients will use $138\mu J$ per stimulation 570 times a day giving a
power consumption of $0.9\mu W$. Similarly, high seizure-activity patients use
$691\mu J$ per stimulation 1330 times a day giving a power consumption of
$11\mu W$ Assuming a battery capacity of $3.3Wh$ [7], the battery for low-
activity patients will last over 14 years, and the battery for high activity
patients will last over 10 years, both of which are significant improvements
from the numbers reported by NeuroPace.

##  VI Conclusions and Future Work

We have demonstrated preliminary results for the implementations of a very-
low-power seizure detection engine which will simultaneously increase
detection accuracies, decrease false positives and over-stimulation, and boost
battery life by up to 50% as compared with state-of-the-art responsive
neurostimulation systems. Each of these improvements will increase a patient’s
quality of life.

Future work will be dedicated toward refining the preliminary results
presented herein including: performing full SPICE simulations of the system
tuned to specific patients, using less-noisy inter-cranial EEG data, and
aiming for a fabricated system. Ultimately, we believe we have demonstrated
the plausibility of our system and shown its potential to improve the quality
of life for patients suffering from epilepsy.

## References

  * [1] W. H. Organisation. (2019, June) Epilepsy. [Online]. Available: https://www.who.int/news-room/fact-sheets/detail/epilepsy
  * [2] aul Boon _et al._ , “Neurostimulation for drug-resistant epilepsy: A systematic review of clinical evidence for efficacy, safety, contraindications and predictors for response,” _Current Opinion in Neurology_ , vol. 31, no. 2, pp. 198–210, apr 2018.
  * [3] E. Bou Assi _et al._ , “Towards accurate prediction of epileptic seizures: A review,” _Biomedical Signal Processing and Control_ , vol. 34, pp. 144–157, 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1746809417300277
  * [4] G. J. Davis P, “Neuromodulation for the treatment of epilepsy: A review of current approaches and future directions.” _Clinical therapeutics_ , June 2020. [Online]. Available: https://doi.org/10.1016/j.clinthera.2020.05.017
  * [5] S. R. Alomar SA, “Different modalities of invasive neurostimulation for epilepsy.” _Neurological Sciences._ , August 2020.
  * [6] J. S. Friedman _et al._ , “Bayesian inference with muller c-elements,” _IEEE Transactions on Circuits and Systems I: Regular Papers_ , vol. 63, no. 6, 2016.
  * [7] NeuroPace. (2020, 6) RNS system physician manual. [Online]. Available: https://www.neuropace.com/wp-content/uploads/2021/02/neuropace-rns-system-manual-320.pdf
  * [8] W.-M. Chen _et al._ , “A fully integrated 8-channel closed-loop neural-prosthetic cmos soc for real-time epileptic seizure control,” _IEEE Journal of Solid-State Circuits_ , vol. 49, no. 1, pp. 232–247, 2014\.
  * [9] M. F. Pinto _et al._ , “A personalized and evolutionary algorithm for interpretable eeg epilepsy seizure prediction,” _Scientific Reports_ , vol. 11, no. 1, p. 3415, Feb 2021. [Online]. Available: https://doi.org/10.1038/s41598-021-82828-7
  * [10] J. Yoo _et al._ , “An 8-channel scalable eeg acquisition soc with patient-specific seizure classification and recording processor,” _IEEE Journal of Solid-State Circuits_ , vol. 48, no. 1, pp. 214–228, 2013.
  * [11] M. A. Bin Altaf _et al._ , “A 16-channel patient-specific seizure onset and termination detection soc with impedance-adaptive transcranial electrical stimulator,” _IEEE Journal of Solid-State Circuits_ , vol. 50, no. 11, pp. 2728–2740, 2015.
  * [12] B. R. Gaines, _Stochastic Computing Systems_. Boston, MA: Springer US, 1969, pp. 37–172.
  * [13] D. H. Hoe, “Bayesian inference using stochastic logic: A study of buffering schemes for mitigating autocorrelation,” _International Journal of Approximate Reasoning_ , vol. 112, pp. 4–21, 2019.
  * [14] S. hua Zhou _et al._ , “An ultra-low power cmos random number generator,” _Solid-State Electronics_ , vol. 52, no. 2, pp. 233–238, 2008. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S003811010700295X
  * [15] K. Yang _et al._ , “A robust -40 to 120°c all-digital true random number generator in 40nm cmos,” in _2015 Symposium on VLSI Circuits (VLSI Circuits)_ , 2015, pp. C248–C249.
  * [16] S. K. Mathew _et al._ , “$\mu$ rng: A 300–950 mv, 323 gbps/w all-digital full-entropy true random number generator in 14 nm finfet cmos,” _IEEE Journal of Solid-State Circuits_ , vol. 51, no. 7, pp. 1695–1704, 2016\.
  * [17] D. Vodenicarevic _et al._ , “Low-energy truly random number generation with superparamagnetic tunnel junctions for unconventional computing,” _Phys. Rev. Applied_ , vol. 8, p. 054045, Nov 2017. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevApplied.8.054045
  * [18] K. Y. Camsari _et al._ , “Stochastic $p$-bits for invertible logic,” _Phys. Rev. X_ , vol. 7, p. 031014, Jul 2017. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevX.7.031014
  * [19] ——, “Implementing p-bits with embedded mtj,” _IEEE Electron Device Letters_ , vol. 38, no. 12, pp. 1767–1770, 2017.
  * [20] V. Shah _et al._ , “The temple university hospital seizure detection corpus,” _Frontiers in Neuroinformatics_ , vol. 12, p. 83, 2018.
  * [21] M. Shoaran _et al._ , “Energy-efficient classification for resource-constrained biomedical applications,” _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , vol. 8, no. 4, pp. 693–707, dec 2018.
  * [22] N. D. Truong _et al._ , “Convolutional neural networks for seizure prediction using intracranial and scalp electroencephalogram,” _Neural Networks_ , vol. 105, pp. 104–111, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0893608018301485
  * [23] H. Daoud _et al._ , “Efficient epileptic seizure prediction based on deep learning,” _IEEE Transactions on Biomedical Circuits and Systems_ , vol. 13, no. 5, pp. 804–813, 2019.
  * [24] J. Yang _et al._ , “From seizure detection to smart and fully embedded seizure prediction engine: A review,” _IEEE Transactions on Biomedical Circuits and Systems_ , vol. 14, pp. 1008–1023, 08 2020.
  * [25] R. Muller _et al._ , “A 0.013 ${\hbox{mm}}^{2}$, 5 $\mu\hbox{W}$ , dc-coupled neural signal acquisition ic with 0.5 v supply,” _IEEE Journal of Solid-State Circuits_ , vol. 47, no. 1, pp. 232–243, 2012.
  * [26] X. Xu _et al._ , “A 1-v 450-nw fully integrated biomedical sensor interface system,” in _2008 IEEE Symposium on VLSI Circuits_ , 2008, pp. 78–79.
  * [27] X. Zou _et al._ , “A 1v 22µw 32-channel implantable eeg recording ic,” in _2010 IEEE International Solid-State Circuits Conference - (ISSCC)_ , 2010, pp. 126–127.
  * [28] T. Denison _et al._ , “A 2.2/spl mu/w 94nv//spl radic/hz, chopper-stabilized instrumentation amplifier for eeg detection in chronic implants,” in _2007 IEEE International Solid-State Circuits Conference. Digest of Technical Papers_ , 2007, pp. 162–594.
  * [29] F. T. Sun _et al._ , “The rns system: responsive cortical stimulation for the treatment of refractory partial epilepsy,” _Expert Review of Medical Devices_ , vol. 11, no. 6, pp. 563–572, 2014, pMID: 25141960. [Online]. Available: https://doi.org/10.1586/17434440.2014.947274