# A New Frontier of AI: On-Device AI Training and Personalization

Jijoong Moon<EMAIL_ADDRESS>0000-0003-0888-2143 ,  Hyun Suk Lee
<EMAIL_ADDRESS>,  Jiho Chu<EMAIL_ADDRESS>,  Donghak Park
<EMAIL_ADDRESS>,  Seungbaek Hong<EMAIL_ADDRESS>,  Hyungjun
Seo<EMAIL_ADDRESS>,  Donghyeon Jeong<EMAIL_ADDRESS>,
Sungsik Kong<EMAIL_ADDRESS>,  MyungJoo Ham<EMAIL_ADDRESS>and
Samsung Research, Samsung ElectronicsSeoulRepublic of Korea

(Date: September 2023)

###### Abstract.

Modern consumer electronic devices have started executing deep learning-based
intelligence services on devices, not cloud servers, to keep personal data on
devices and to reduce network and cloud costs. We find such a trend as the
opportunity to personalize intelligence services by updating neural networks
with user data without exposing the data out of devices: on-device training.
However, the limited resources of devices incurs significant difficulties. We
propose a light-weight on-device training framework, NNTrainer, which provides
highly memory-efficient neural network training techniques and proactive
swapping based on fine-grained execution order analysis for neural networks.
Moreover, its optimizations do not sacrifice accuracy and are transparent to
training algorithms; thus, prior algorithmic studies may be implemented on top
of NNTrainer. The evaluations show that NNTrainer can reduce memory
consumption down to 1/20 (saving 95%!) and effectively personalizes
intelligence services on devices. NNTrainer is cross-platform and practical
open-source software, which is being deployed to millions of mobile devices.

On-Device AI, Neural Network, Personalization, Training, Software Framework

††conference: 46th International Conference on Software Engineering; April
2024; Lisbon, Portugal

##  1\. Introduction

We have witnessed the rapid proliferation of deep neural networks for a wide
range of consumer electronics products in the industry. Their intelligence
services provide key features to consumer electronics: semantic segmentation
(Pal and Pal, 1993) for smartphone cameras, super-resolution (Yang et al.,
2019) for TVs, object detection for robotic vacuums, image classification
(Krizhevsky et al., 2017; Wu et al., 2015) for smart ovens, speech recognition
(Dahl et al., 2011; Hinton et al., 2012) for smartphones and TVs, and ASR
(Ling et al., 2015) and TTS (Qian et al., 2014) for real-time translations.
The quality of such services has become significantly important for consumer
electronics; thus, a lot has been invested to develop and optimize deep neural
networks.

Since the proliferation of intelligence services in consumer electronics, the
need for adaptation to different environments and requirements of each
individual user, “personalization”, has arisen with additional technical
difficulties (Justin Basilico, 2021). Because we use general data to train
neural network models for the general public, not personal data for a personal
model, for an individual user, the model may look over-parameterized or its
quality of service is low compared to the size of the model.

To personalize intelligence services, we target to update models on devices
with data available on devices: “on-device training”. Note that we do not need
to train the whole models on devices; with personal data, we can update, fine-
tune, or adjust models pre-trained by general data. We may increase the
accuracy of a user with the user’s data. We may add classes defined by a user
to personalize the service for the user. We may reduce latency and energy
consumption for a user by reducing or skipping over-parameterized parts
(Teerapittayanon et al., 2016).

Training models on devices instead of clouds provides significant advantages.
By running intelligence services on devices, “on-device AI” (MIT Technical
Review, [n. d.]; Ham et al., 2021), we can save cloud operating costs (we have
hundreds of millions of active mobile phones and consumer electronics
deployed!), and we can keep personal data and privacy without exposing them
externally. By training on devices, not clouds, we can achieve the same
advantages. Besides, regulations and consumers’ expectations on personal data
and privacy are becoming more strict; e.g., the General Data Protection
Regulation (GDPR) (Voigt and Von dem Bussche, 2017) makes it extremely
difficult to gather personal activity records in clouds. A frequently used
device (e.g., a mobile phone) usually generates personal data continuously,
which can enable continuous personalization with online or continual learning
techniques (Delange et al., 2021; Thrun and Mitchell, 1995; Shin et al., 2017)
on the device. Federated learning (Li et al., 2020; Essalmi et al., 2010)
usually requires on-device training mechanisms, too.

Limited resources of devices pose challenges to training on devices especially
if the accuracy cannot be sacrificed for resources; i.e., business units
usually prohibit such trade-off in the authors’ affiliation. For example,
studies to reduce the memory overhead emerge as larger neural networks become
popular: dynamic sparse reparameterization (Mostafa and Wang, 2019), low
precision training (Narang et al., 2018), reduced batch sizes(Huang et al.,
2019), and gradient checkpointing (Bulatov, 2018; Chen et al., 2016), which
address algorithmic aspects of models. However, the system software aspect for
training optimization is seldom studied, and we can improve significantly by
addressing the structure and complexity of training software implementations.
We address the system software aspect (i.e., how memory and computation are
allocated and scheduled) to optimize, which has been often overlooked in
neural network frameworks. Our approach does not sacrifice accuracy to
conserve resources, which is often prohibited by applications. Moreover, the
algorithmic improvements mentioned above can be applied transparently on top
of the proposed mechanism.

We address the execution orders of training procedures based on the
observation that we can statically calculate computation and memory
requirements from the model structure. Thus, by controlling such execution
orders, we can optimize resource utilization. First, we divide a training
session into fine-grained procedures and determine the life cycles of memory
blocks assigned for procedures so that controlling execution orders of the
procedures can determine the memory consumption accordingly. Then, we identify
possible execution orders for each neural network layer type and schedule them
to minimize peak memory consumption. Finally, we apply another optimization
technique, Proactive Swap; i.e., we know when a buffer is read or written
(clairvoyant!); thus, we can swap in and out proactively, minimizing its
performance impacts.

Our contributions can be summarized as follows:

  * •

We propose a highly memory-efficient on-device training technique without
sacrificing accuracy and latency, exploiting the novel observation on the
nature of neural network training, and Proactive Swapping. The peak memory
consumption is dramatically reduced, realizing training on embedded devices.
Moreover, the techniques are general; conventional machine learning frameworks
may adopt them for servers, and neural network algorithmic optimization may be
applied simultaneously.

  * •

We implement and release the proposed techniques as an open source, cross-
platform, and commercialization-ready framework, NNTrainer, which is already
applied to actual products.

  * •

We evaluate NNTrainer with various models and conventional frameworks, and
show that it is highly efficient and effective. We demonstrate that complex
encoder-decoder models based on Tacotron2 (Shen et al., 2018) and Transformer
(Vaswani et al., 2017) can be personalized with sufficient batch sizes on
mobile phones, which are deployed for products in early 2023.

##  2\. Related Work

With larger neural networks being popular, studies on algorithms to consume
less resources to train have been published, which usually trade-off between
resources and accuracy with neural network algorithmic approaches. A study
(Mostafa and Wang, 2019) proposes dynamic sparse reparameterization to reduce
the memory requirements by making the weight and activation values sparse
during training. It generates smaller model sizes that incur less peak memory
consumption and computation with the sparsity by sacrificing the accuracy. By
adopting 16 bits float precision instead of 32 bits, (Narang et al., 2018)
reduces the memory and computation requirements, which also reduces the model
size; however, it also sacrifices the accuracy. The activation values occupy a
significant part of memory and microbatching technique (Huang et al., 2019)
helps reduce the size of such values. However, it alters the statistical
properties of batch normalization, which affect the accuracy. There are
studies (Bulatov, 2018; Chen et al., 2016) to reduce memory consumption
without sacrificing accuracy by storing the activation values partially and
recomputing the activation values not stored; however, obviously, this
increases the computation cost by about 30%. ZeRO-Offload (Ren et al., 2021)
and ZERO-Infinity (Rajbhandari et al., 2021) propose software optimization
mechanisms without sacrificing the accuracy by swapping the memory block to
the secondary storage. However, unlike (Ren et al., 2021; Rajbhandari et al.,
2021), which swaps in and out reactively, the swapping mechanism of NNTrainer
swaps in and out according to the execution order; i.e., the swapping becomes
proactive, which minimizes performance deterioration. Another difference is
that they focus on offloading the sliced chunks of activation or weight; thus,
it is difficult for (Ren et al., 2021; Rajbhandari et al., 2021) to support
gradient clipping or non-trainable layers.

There are several popular software frameworks to train the neural network
model: TensorFlow (Abadi et al., 2016) and PyTorch (Paszke et al., 2017).
TensorFlow is a popular open-source machine learning framework from Google. It
includes Keras and various tools for developers. A lightweight variant of
TensorFlow, called TensorFlow-Lite (Google, 2018), targets on-device AI, and
training features have been enabled recently. It is written in C/C++ rather
than Python. However, unlike NNTrainer, developers cannot update model
structures with the lightweight TensorFlow-Lite on device; the full TensorFlow
is required for such tasks.

PyTorch (Paszke et al., 2017), including Caffe2 (Jia et al., 2014), is another
popular open-source machine learning framework from Facebook, which includes
highly intuitive APIs. It has an experimental lightweight variant, PyTorch
Mobile, which runs mobile models converted from PyTorch models. Note that
PyTorch Mobile targets inference and does not support training.

TensorFlow, TensorFlow-Lite, and PyTorch consume an excessive amount of memory
to train models (shown in §5); thus, consumer electronics cannot use them to
train models. For example, the quality assurance team of the authors’
affiliation will drop in-house applications consuming an excessive amount of
memory because the main memory may be crowded by a large number of
applications preloaded (Lim et al., 2023). Note that reducing the memory
consumption is beneficial for conventional server-based machine learning;
i.e., we may increase the batch size with less cost.

##  3\. Analysis

Figure 1. Memory buffer usage of forward and backward processes.

This section elaborates on the observation and analysis of the neural network
training processes and their resource requirement and management. Figure 1
describes the memory usage for forward and backward processes. Let’s denote
the network $T=\\{Y,G(X_{0})\\}$, where
$G(X_{0})=G_{n}(G_{n-1}(...G_{1}(G_{0}(X_{0}))))$ as a sequence of layers and
$Y$ is label.

In a forward process of a layer, $L_{i}$, in Figure 1.a takes $X_{i}$ as an
input, calculates $G_{i}$, and saves the output, ${X_{i}}^{\prime}$. $X_{i}$
and ${X_{i}}^{\prime}$ are saved for the gradient calculation of backward
operations. Unlike a forward process, a backward process starts from the last
layer, where the loss is calculated and propagated. A backward process
calculates the gradient $\Delta W_{i}$ with the input derivative
$\Delta{D_{i}}$ and $X_{i}$ saved in a forward process, and updates the weight
$W_{i}$ using an optimizer. Then, it calculates the output derivative,
$\Delta{D_{i}}^{\prime}$, to propagate the input derivative $\Delta{D_{i}}$ to
$L_{i-1}$.

Depending on the layer type, a layer requires buffers for weight $W_{i}$,
gradient $\Delta W_{i}$, input $X_{i}$, and output $G_{i}(X_{i})$. Usually,
the sizes of such buffers depend on the input size and the depth of the neural
network. When the input size and the network configuration are determined, the
amount of memory required for training can be roughly calculated. Assume that
there is a convolution 2D layer with the same padding, $1\times 1$ stride, and
64 filters with $3\times 3$ size, which takes an input image of $32\times
32\times 3$. Then, the input size is 0.39 MiB for 32 batch size (width
$\times$ height $\times$ channels (3) $\times$ sizeof (float) $\times$ batch
size), and the output buffer size is 8.3 MiB (width $\times$ height $\times$
channels (64) $\times$ sizeof (float) $\times$ batch size). We also need
derivatives ($\Delta D_{i}$, $\Delta D_{i+1}$), weight ($W_{i}$) and gradient
($\Delta W_{i}$) for backward process. Therefore, we need about 16.6 MiB of
heap memory to train a single layer. For “deep” neural networks, we need
multiple times, often over a hundred times, of such an amount of memory
depending on the depth of the given neural network model.

Figure 1.b shows the memory buffers for neural network training. It repeatedly
reuses the memory buffer configuration of a single layer depicted in Figure
1.a, which is equivalent to dotted rectangles in Figure 1.b. Conducting
forward and backward processes for $n$ layers requires memory space of about
$n/2$ times the memory space required for a single layer. Because the output
of $L_{i}$, ${X_{i}}^{\prime}$, and the input of $L_{i+1}$, $X_{i+1}$,
represent the same data and layers do not modify input data, the two may share
the same buffer instance. Besides, we can reduce memory further as shown in
Figure 1.c. Although the output of each layer, ${X_{i}}^{\prime}$, is stored
for the backward process, its derivatives, $\Delta D_{i}$ and $\Delta
D_{i}^{\prime}$, are not required after the completion of the layer’s backward
process. Therefore, memory space for the derivatives can be shared. The same
optimization can also be applied to the gradients because they are not
required once the weight is updated with the gradients.

There are layers that may reuse input buffers, $X_{i}$, as output buffers;
e.g., activation layers. For example, let ${X_{i}}^{\prime}$ be the output of
a sigmoid activation function, then, its derivative is
$\Delta{D_{i}}^{\prime}={X_{i}}^{\prime}(1-{X_{i}}^{\prime})$. During a
backward process, computing $\Delta{D_{i}}^{\prime}$ requires
${X_{i}}^{\prime}$, which is the output of a corresponding forward process,
not the input of it. Therefore, only one intermediate activation is required
to store the output ${X_{i}}^{\prime}$; i.e., an in-place computation shown as
$L_{3}:InplaceOp$ in Figure 1.c. This allows for freeing the memory space
storing inputs of activation layers. Because activation layers are applied
after most operations, including convolution and linear layers, this method
reduces the memory requirement of inputs by almost half. This can be applied
to batch normalization as well.

Figure 2. Different granularity of training procedures. A gradient needs to be
computed before the derivative; otherwise, $X$ remains during derivative
computation. Figure 3. Different types of training procedures.

From the analysis, we find that different granularity of training procedure is
possible, reducing the memory consumption with its own pros and cons as in
Figure 2. Coarse-grained procedure as in conventional training frameworks
(TensorFlow and PyTorch) costs less to develop a new layer because only its
forward process is required to be implemented with Automatic Differentiation
(Baydin et al., 2018); however, it is difficult to systematically find the
minimum memory requirement. On the other hand, fine-grained training
procedures can clearly distinguish such as forwarding, calculating gradient,
calculating derivative, and applying gradient; thus, the activation and
derivative memory, $X$ and $\Delta D$ are not required at the same time, so
that we can minimize the memory consumption further. NNTrainer has fine-
grained training procedures implemented by assigning the execution order (EO)
to each procedure, and it has scheduling algorithms(§4) based on EO. With the
scheduling algorithms, NNTrainer overwrites invalid memory buffers such as $X$
after computing gradients to avoid de-allocating and re-allocating buffers. It
also helps to reduce I/O interactions during the memory-to-storage swapping.
Prior art (Bulatov, 2018; Chen et al., 2016; Shah et al., 2020) mainly focuses
on managing the activation memory computed at the forwarding procedure. Unlike
prior art, we find more opportunities during the backward procedures and
propose a highly efficient memory optimization scheme without sacrificing
accuracy. Therefore, previous works can be applicable on top of our work.

The resource optimization schemes should be applicable for various scenarios
including examples in Figure 3. The proposed mechanism supports a mixed
network of trainable layers and non-trainable layers, which is common in on-
device AI personalization (Torrey and Shavlik, 2010; Lu et al., 2015; Tan et
al., 2018). In non-training layers, forwarding processes do not need to store
$X_{0}$ because of skipping gradient-related processes shown in Figure 3.b.
Although conventional frameworks support non-trainable layers, they still
allocate memory for activation inputs, $X_{0}$. We evaluate with a fully
connected layer having 131,584 parameters (514 kiB of float32) in the middle
of a model with GPUs and confirm that the memory of gradient is reduced as in
Figure 3.b. Moreover, the re-ordering of fine-grained training procedures
enables the Gradient Clipping (Zhang et al., 2020) easily as in Figure 3.a.
Therefore, the fine-grained training procedure is more robust and efficient
for supporting various training processes and resource utilization
optimizations.

##  4\. Design

This section describes the design and implementation of NNTrainer. NNTrainer
is a modular framework written in C++ with many user-extensible components,
whose structure is shown in Figure 4. It is cross-platform; we release to
Ubuntu, Tizen, Android, and Windows.

Figure 4. Abstract architecture of NNTrainer.

The training processes of NNTrainer can be categorized as Load, Configure,
Compile, Initialize, setData, and Train, in the order we elaborate.

Load: the output of Load is an instance of Model, which has all the
information of a neural network model, including hyper-parameters, and
orchestrates all the components and processes. Users may describe a neural
network as an instance of Model with an initialization file (ndevilla, 2017)
or APIs in C, C++, .NET, and Web. The interpreter in Model Loader parses model
descriptions and generates intermediate graph representations. Developers may
extend Interpreters to support model files of other frameworks; we provide an
extension for TensorFlow-Lite (Abadi et al., 2016). The output of Load
describes layers of the model as tuples of [<Layer type>, <Properties (key,
value)>].

Configure creates layer objects with those tuples, and the layer objects
construct a Graph. We construct Graph and Layer instances with delayed
allocation based on the information provided by Context, which has parameters
of Graph and Layer instances. It conserves memory by delaying Tensor buffer
allocations until the buffers are actually required.

Compile: before Initialize, Layer Context exists as an initContext instance
storing the information as strings. Later, initContext becomes runContext by
Initialize process, converting string descriptions to Tensor objects.

It is lighter and more efficient to conduct the Compile process with
initContext; it handles strings, not Tensor instances. Each Layer subclass
provides forward and backward functions that calculate gradients and
derivatives, respectively. Each Optimizer subclass provides a function
applying gradients. Users can implement new layers and optimizers by adding
these subclasses.

For higher memory efficiency, we need to manage Tensor data independently;
thus, we have separated specifications (dimensions and status) and data stored
in buffers.

Realizer | Role
---|---
Flatten | Identify and create a flatten layer
Activation | Identify and create an activation layer
BatchNorm | Identify and create a batch normalization layer
Multi-Out | Identify and make connection to layers
Loss | If loss is cross entropy, remove the activation
Concat | Identify inputs and create Concatenate layer
Recurrent | Unroll the graph if there is a loop
Slice | Create sub-graph network in the backbone model
Input | Identify and create an input layer

Table 1. Default Realizer subclasses.

The Model’s Graph, created by Configure process, can operate after Realizer of
Compiler applies lowering operations; i.e., the Realizer adds layers or
changes the order based on the analysis of graph structures. Table 1 shows the
default Realizer subclasses of NNTrainer. Compile process computes connection
and ordering with the provided Graph Optimizer subclasses, which are critical
for memory saving. Users may extend both Graph Optimizer and Realizer by
adding subclasses.

Initialize: in Initialize process, a finalize method of a Layer subclass is
executed while visiting each Layer instance in Graph, and Tensor instances are
requested to Tensor Pool. Because specification and data of Tensor are managed
independently, NNTrainer manages memory separated by Tensor Pool and Memory
Pool. After Tensor Pool creates all requested Tensors instances throughout the
Model, Memory Planner plans with Tensor instances given by Tensor Pool,
computes the offset of the Memory Pool for each Tensor instances, and assigns
the memory for Tensor data. NNTrainer provides the Memory Swap using Cache
Pool and Loader. Loader creates an independent Task Executor and monitors the
existence of the tensor data in memory. It loads and unloads proactively using
execution order (EO) to hide the data transfer overhead (§3).

setData: to generate batch-sized data for training in setData process,
NNTrainer provides DataSet, and users can provide training data using
DataProducer, which is extendable. DataProducer generates data for training
and accumulates the data in the Batch Queue up to the batch size. After the
described processes, we are finally ready for Train process.

In terms of resources, the uniqueness of NNTrainer over the conventional is
that Tensors in NNTrainer are prioritized based on EO and in-place operation.
This allows Memory Planner to maximize the reusability of Memory Pool; thus,
reducing memory consumption.

###  4.1. Temporal and Spatial Relation

Temporal Specification | fine-grained processes
---|---
Forward (F) | forward
Compute Gradient (CG) | compute gradient
Compute Derivative (CD) | compute derivative
Apply Gradient (AG) | apply gradient
Backward (B) | compute gradient, derivative and apply gradient
Iteration (I) | reset after iteration
Max (M) | always available

Table 2. Temporal relations.

During training, intermediate activation accounts for more than 90% of the
total memory consumption (Cai et al., 2020), and forward processes decide
intermediate activation values, which are saved for backward processes to use.
Thus, if EOs effectively exploit orders between processes, we can use memory
more efficiently as in Figure 2.b. To achieve this, NNTrainer divides the
entire training process into Forward, Compute Gradient, Compute Derivatives,
and Apply Gradient, and configures EOs of each layer. It is straightforward to
determine the orders of layers; however, memory scheduling and swapping
require more information.

To address memory scheduling, NNTrainer defines temporal relations (Table 2)
and spatial relations (Table 3) in Tensor specification. Spatial relation is
assigned automatically during Initialize process by analyzing the network
graph. If a buffer for intermediate activation is requested, Create(C) is
assigned by default. However, if its previous layer is an in-place operation,
Modify View (MV) or Read-Only View (RV) is assigned, determined by the
behavior of the given layer for the memory buffer, and if the previous layer
is ReLU or sigmoid, NNTrainer assigns Modify View (MV). If the examined layer
is flatten or reshape, which modifies the dimensions of tensors, not their
contents, then Read-Only View (RV) is assigned. Extend (E) creates tensor
sharing everything; e.g., weight tensors of an unrolled sub-graph for the
recurrent network.

Dependency | Spatial Relation | Description
---|---|---
Independent | Place-Holder (P) | Holding the external memory
Create (C) | Allocate new Tensor
Memory sharing | Modify View (MV) | Data changes
Read-Only View (RV) | No data change
Tensor sharing | Extend (E) | Share everything

Table 3. Spatial (inter-layer) relations. Algorithm 1 Compute Execution Order.

Input: Layer List: ${L=\\{L_{0},L_{1},...,L_{N-1}\\}}$

1:$EO_{max}=N\times 4$

2:Initialize $T_{map}$

3:for $i=0,1,\ldots,N-1$ do

4: $EO_{F}=i$

5: $EO_{CG}=N+i\times 3$

6: $EO_{AG}=EO_{CD}+1$

7: if Gradient Clipping then

8: $EO_{CG}=N+i\times 2$

9: $EO_{AG}=N\times 3+i$

10: end if

11: $EO_{CD}=EO_{CG}+1$

12: if Non-Trainable then

13: $EO_{CG,CD,AG}=N+i\times 3$ $\triangleright$ Set $EO_{L_{i}}$ same & run
compute derivatives only

14: end if

15: $EO_{L_{i}}=\\{EO_{F},\ EO_{CG},\ EO_{CD},\ EO_{AG}\\}_{Layer\ Type}$

16: for $t_{r}\in T_{L_{i}}$ do $\triangleright$ $T_{L_{i}}$ is Tensor list
requested in $L_{i}$

17: $t_{r}=T_{map}.get(t_{r})$ or $T_{map}.insert(t_{r})$

18: Add $EO\in EO_{L_{i}}$ to $t_{r}$ w.r.t Temporal of $t_{r}$

19: end for

20:end for

21:Sort $T_{map}$ by ascending with $min(EOs\ of\ T_{i}\in T_{map})$

22:for $T_{i}\in T_{map}$ do

23: $CM_{T_{i}}$ = Tensor create mode of ${T_{i}}$

24: if $CM_{T_{i}}==MV_{T_{j}}$ then $\triangleright$ $T_{j}$ is target Tensor

25: if $min(EOs\ of\ T_{i})\geq max(EOs\ of\ T_{j})$ then

26: Merge $T_{i}$ to $T_{j}$ including $EOs$

27: end if

28: else if $CM_{T_{i}}==RV_{T_{j}}$ or $E_{T_{j}}$ then

29: Merge $T_{i}$ to $T_{j}$ including $EOs$

30: end if

31:end for

Algorithm 1 describes how EO is defined using temporal and spatial relations
of tensors. Figure 5 shows how Algorithm 1 works for an exemplar model of
three layers, where $L_{1}$ is a sigmoid activation and $L_{2}$ is a flatten
layer. In the figure, $0,9(F,CG/P)$ denotes that the temporal relation is $F$
and $CG$, the spatial relation is $P$, and the EOs are 0 and 9. It
sequentially assigns EOs starting from $L_{0}$, and the result is shown in the
bottom-right square in the figure.

In $L_{0}$, requested tensors are $X_{0}$, $X_{1}$, $D_{1}$, $\Delta W_{0}$,
and $W_{0}$. Because the temporal relations of $X_{0}$ are $F$ and $CG$, their
EOs, 0 and 9, are assigned to $X_{0}$. Then, set 0 ($F$ of $L_{0}$) for
$X_{1}$ and set 9 ($CG$ of $L_{0}$) for $D_{1}$. This procedure is repeated
for each Layer, $L_{i}$: line 3 to 20 in Algorithm 1. After merging output
tensors, $X_{i-1}^{\prime}$ and $X_{i}$ in Figure 1, we can calculate
theoretical memory requirement of the given model, which provides the basis
for peak memory consumption comparison in Evaluation.

Figure 5. Execution orders and temporal-spatial relations of an example model,
where only $X_{0}$, $X_{1}$, $D_{3}$, $\Delta W_{0}$, and $W_{0}$ are
required. Refer to Figure 1, Table 2 and 3 for the notations.

Spatial relations appear between an in-place layer and its adjacent layer as
in Figure 5. Then, the input, $X_{1}$, and output, $X_{2}$, of $L_{1}$ can be
Memory Sharing (Table 3) with changing data, and $X_{2}$ can be marked as
Modify View of $X_{1}$ ($MV_{X_{1}}$). Read-Only View ($RV$) of $X_{2}$
($RV_{X_{2}}$) can be given to $X_{3}$.

For a given spatial relation of $L_{1}$, let’s call $X_{1}$ a target Tensor
and $X_{2}$ a merged Tensor. Similarly, $X_{2}$ is a target Tensor for a
merged Tensor $X_{3}$ for a spatial relation of $L_{2}$. When applying spatial
relation, only the largest order of the target Tensor and the smallest order
of the merged Tensor are compared: line 25 of Algorithm 1. The largest order
of $X_{1}$, 1, is equal to the smallest order of the merged Tensor, $X_{2}$,
1; thus, these Tensors and orders can be merged as 0, 1, 2, 7. If the largest
order of the target Tensor is greater than the smallest order of the merged
Tensor, the integrity of the target Tensor cannot be guaranteed because the
target Tensor is accessed after the merge; thus, it cannot be combined and a
new Tensor needs to be created. Next, the largest order of $X_{1}+X_{2}$ is
greater than the smallest order of the merged Tensor, $X_{3}$, 2; however, it
can still be merged because the integrity of data is guaranteed with RV. This
reduces memory requirement by removing another intermediate activation as
described in Figure 1.

###  4.2. Memory Pool and Memory Planner

With EOs assigned, Memory Planner allocates buffers from Memory Pool with the
planner algorithm, Algorithm 2. Algorithm 2 is simple sorting-based; an
algorithm minimizing fragmentation for higher utilization is future work.
Figure 6 shows how Algorithm 2 works for a case without spatial relations as a
simple example, where the four iterations of the for-loop, $i=[7...10]$, are
shown, reallocating a few tensors to share and reuse the memory space.

Algorithm 2 Simple sorting-based Memory Planner.

Input: Tensor List: ${T=\\{T_{1},T_{2},...,T_{N}\\}}$

1:Sort T by ascending order based on $min(EOs\ of\ each\ T_{i})$

2:if $min(EOs\ of\ T_{i})$ == $min(EOs\ of\ T_{j})$ then

3: Sort by descending order based on $max(EOs\ of\ T)$

4:end if

5:for $i=1,2,\ldots,N$ do $\triangleright$ ${T}$ is Sorted, now

6: ${EO_{i}}^{min}$ = $min\left(EO\ of\ {T_{i}}\right)$

7: $last=i$

8: for $j=i-1,\ldots,1$ do

9: ${EO_{j}}^{max}$ = $max\left(EO\ of\ {T_{j}}\right)$

10: if ${EO_{j}}^{max}<{EO_{i}}^{min}$ then

11: $last=j$

12: end if

13: end for

14: if ${last}=={i}$ then

15: Calculate offset for ${T_{i}}$ and Assign to ${T_{i}}$ data

16: else

17: Assign data(offset) of ${T_{last}}$ to ${T_{i}}$ data(offset)

18: end if

19:end for

Figure 6. Memory planning without spatial relation for three linear layers.

For each Tensor data, we calculate the memory offset to assign. In Figure 6,
when we calculate the offset of $W_{1}$ after calculating the offset of
$W_{0}$, because there are no assigned Tensors whose largest EO is less than
the smallest EO of $W_{1}$, a new offset is calculated and assigned to
$W_{1}$. It keeps assigning new offsets until $X_{3}$ as shown in the first
row of Figure 6. As in the second row of the figure, when we calculate for
$D_{3}$, the largest EO of $X_{3}$, 2, is less than the smallest EO of
$D_{3}$, 3. It means that $X_{3}$ can be reused; thus, the same offset is
assigned to the data of $D_{3}$ as in line 17 of Algorithm 2. To assign
$\Delta W_{2}$ in the next step, a new offset is required because there is no
Tensor whose largest EO is smaller than the smallest EO of $\Delta W_{2}$.
Memory planning is completed when calculating offsets and assigning Tensors
are completed.

A major advantage of this method is that we can calculate the peak memory
consumption beforehand as shown in Figure 6, which is equal to the ideal
memory requirement in Figure 1. Even with cloud servers and workstations, out-
of-memory is often the roadblock of machine learning tasks, which is even more
critical in mobile and embedded devices. By calculating the peak memory
requirement beforehand, engineers can plan machine learning tasks more
effectively and try more diverse hyper-parameters, different model structures,
or higher resource utilization (e.g., increased batch sizes).

###  4.3. Reduced and Proactive Swap

Figure 7. On-demand Swap (5 linear layers). Figure 8. Reduced and Proactive
Swap (Lookahead 1).

Most of the previous works focus on reducing the intermediate activation which
occupies a huge part of memory during training. In order to resolve this
problem, NNTrainer proposes the Proactive Swap scheme on top of NNTrainer
memory planning based on EO. To schedule swapping, the validity of tensors,
which differs for each training procedure, should be considered. The weight
tensors should be valid throughout the whole training procedure without any
allocation and deallocation (Light blue area in Figure 7), and the input and
output tensors for activation should be valid until gradient computation whose
inputs are the results of forwarding computation (Light green area in Figure
7). NNTrainer uses individual I/O streams for each Tensor Pool Stream (TPS)
and Weight Pool Stream (WPS) considering different characteristics of tensors.
The simplest swap scheme is On-Demand in Figure 7 which conducts swap-in/out
when required without any concern about EOs.

Figure 8 shows an example of how Reduced and Proactive Swap works with three
linear layers. In Figure 8.a, the memory scheduler of NNTrainer counts
temporal and spatial relations, and reduces the number of swapping by reusing
input tensors ($X_{3}$) for input derivatives ($D_{3}$) between $CG_{3}$ and
$CD_{3}$. Figure 8.b shows a Proactive Swap example with +1 lookahead from
$AG_{3}$ to $CD_{2}$; Proactive Swap at the n’th EO hides I/O overhead by pre-
loading tensors of n+lookahead EO and offloading tensors of n-1’th. For each
EO, an I/O stream swaps out tensors used by the previous EO and swaps in
tensors for the next (lookahead +1) EO. We can reduce the swap overhead
further by merging conflicting swap requests; e.g., $D^{\prime}_{3}$ and
$D_{3}$ at $AG_{3}$. Then, we can compute the memory overhead of Proactive
Swap. Tensors in memory at $AG_{3}$ are $X_{3},D^{\prime}_{3},\Delta W_{2}$,
and $W_{3}$ after Proactive Swap. Only $W_{3}$ is the memory required
additionally from the on-demand swap, and it is the bearable cost for the
reduced latency.

The lookahead is a hyper-parameter determined statically at the
initialization, and its default value is one. However, the optimal lookahead
differs by network configurations and execution orders; e.g., gradient
clipping, distribution of non-trainable layers, and layer types. Thus, we can
further reduce memory and computation overhead by assigning lookahead for each
EO dynamically; i.e., let each lookahead value converge throughout iterations.
This is left for future work.

##  5\. Evaluation

We evaluate both small experimental neural networks and large practical neural
networks. We analyze and compare latency and memory consumption of NNTrainer
and conventional frameworks: PyTorch 1.13.1, TensorFlow 2.11.0, TensorFlow-
Lite 2.11.0 (both C++ and Python implementations); recent TensorFlow-Lite
supports training. For memory consumption, we also compare with the
theoretical memory requirement based on Analysis (§3). We demonstrate on-
device training applications on devices: a complex text-to-speech (TTS)
application with multiple LSTM layers on Galaxy S21 Ultra (Exynos 2100,
1$\times$ARM Cortex-X1 2.9 GHz, 3$\times$ARM Cortex-A78 2.9GHz,
4$\times$Cortex-A55 2.2GHz) and a Transformer model with huge memory
requirements, virtually impossible for embedded devices to run. Except for
TTS, we evaluate with Raspberry pi 4, which has 4$\times$ARM Cortex-A72
(1.5GHz) CPU cores and 8 GiB RAM running Ubuntu 22.04. We evaluate NNTrainer
with CPUs as its computation backend. Note that most NPUs and DSPs of embedded
devices do not support floating points and GPUs of consumer electronics are
often not available for training; e.g., GPUs of TV are supposed to be fully
dedicated to video streams and GUIs.

###  5.1. Component Evaluation

Table 4 describes 5 test cases of small neural networks with different
dimensions of inputs and labels. The test cases include an LSTM layer (popular
for time series including voice models) and linear and Conv2D layers (popular
for vision models). We evaluate a case of three linear layers (FC-FC-FC) with
non-trainable layers in the middle and a case of Figure 5 with Conv2D layers
(Conv-AC-FL). We have applied mean squared error (MSE) (Allen, 1971) and
stochastic gradient descent (SGD) (Ketkar, 2017) to train the test cases.

We ensure the correctness of NNTrainer by comparing every activation and
weight value of models trained by NNTrainer with the same models and data
trained by TensorFlow. The two frameworks result in equivalent neural network
models with errors at $10^{-4}$ level. Note that an automated test suite (Lim
et al., 2021) ensures the correctness and equivalence for each pull request in
different environments.

Test Cases | Input | Output (Label) | Theoretical Memory (KiB)
---|---|---|---
Linear | 64:1:1:150528 | 64:1:1:300 | $390590$
Conv2D | 64:3:224:224 | 64:3:112:112 | $65856$
LSTM | 64:1:1:150528 | 64:1:1:10 | $84731$
FC-FC-FC | 2048:1:1:784 | 2048:1:1:100 | $135928$
Conv-AC-FL | 64:3:224:224 | 64:1:1:37632 | $65856$

Table 4. Test cases for component evaluation.

As explained in Section 3, once the input sizes and configurations are
determined, the theoretical memory requirements can be calculated based on the
sizes of intermediate activations, weights, and gradients as in Table 4.

Figure 9. Peak memory consumption Figure 10. Non-trainable layer analysis

Figure 9 shows the evaluation results of the baseline and peak memory
consumption. The baseline implies memory consumed by the framework itself
(e.g., codes and libraries), which is constant for different test cases. The
peak implies the memory consumed by training a model in addition to the
baseline. TensorFlow consumes x27 of baseline memory (329.8 MiB) than
NNTrainer (11.3 MiB) consumes, and PyTorch consumes x8 (103.4 MiB).
TensorFlow-Lite in C++ (39.6 MiB) consumes three times. Such excessive
baseline memory consumption of TensorFlow, TensorFlow-Lite(python) and PyTorch
can be attributed to Python and a lot of external libraries; NNTrainer and
TensorFlow-Lite (C++) are written in C++ with minimal dependencies on external
libraries. The peak memory consumption varies per test case, and, unlike the
baseline, the difference is mostly caused by design choices (including
granularity of training procedures) and buffer management approaches affected
by the choices, not by the libraries and languages for the implementation.

In single and multi-layer tests of Linear, Conv2D, and LSTM, NNTrainer
consumes significantly less amount of memory. The proposed resource
utilization method based on EOs allows reusing memory spaces for Tensors
between layers. Note that conventional frameworks consume significantly larger
memory than NNTrainer does: x2.52 to x11.47 on average including baseline.
Even Tensorflow-Lite in C++ consumes x4 of that NNtrainer consumes. The
theoretical memory requirement in Figure 9 suggests that NNTrainer is
extremely efficient in using memory by sharing the most shareable tensors with
ignorable memory overhead. The baseline is required to load essential
libraries, and additional heaps attributing to the peak are required for some
layers; e.g., NNTrainer’s Conv2D layer adds “Image to Column” (im2col)
(Dukhan, 2019) operator for computation efficiency, which requires additional
buffers.

We evaluate how much memory can be reduced by NNTrainer with FC-FC-FC case by
setting the second layer non-trainable. Figure 10 shows the memory placement
along with EOs by NNTrainer’s memory planner. $X_{1}$ is the input tensor of
the second layer which needs to be saved for computing gradient ($L_{1}:CG)$
for the training case (Figure 10.a). For the non-trainable case, the validity
of $X_{1}$ is during forwarding as in Figure 10.b so that the memory space is
utilized by other tensors. The results show the memory reduction at $L_{1}:CG$
(red rectangle), the non-trainable layer. However, the peak memory is measured
at a different EO and 24.4 MiB is expected to be reduced. The experimental
results show that 20.9 MiB is reduced: 146.1 MiB vs. 125.2 MiB.

Figure 11. Training latency of the test cases from Table 4. Figure 12. VGG16
training performance of swap schemes. On-Demand and Reduced show the exactly
same performance.

Figure 11 shows the training latency of 1 epoch, 640 dataset size with the
test cases in Table 4. This experimental result shows that NNTrainer does not
sacrifice the latency or the accuracy to conserve memory. Although consuming
significantly less memory, in most cases, NNTrainer is evaluated to be faster
than or equivalent to the conventional frameworks.

###  5.2. On-Demand, Reduced, Proactive Swap

Figure 13. Latency of Proactive Swap (VGG16)

Swapping allows additional memory saving selectively; users may turn it on and
off. We compare the peak memory consumption and the number of swapping for
different schemes, On-Demand, Reduced, and Proactive Swap with one lookahead
parameter, in Figure 12. Proactive Swap includes Reduced Swap. We train the
VGG16 model with 64 batches of $32\times 32\times 3$ inputs. Figure 13 shows
the latency of each EO for each scheme. As expected, without swapping, the
largest size of memory is consumed, although it is still the smallest among
the compared frameworks in Figure 14. The evaluations show that Proactive Swap
reduces the peak memory consumption from 181 MiB to 71 MiB successfully with
20.1% latency overhead and On-Demand Swap reduces to 51 MiB with 35.5% latency
overhead. The peak memory consumption of On-Demand Swap is 59 MiB, the same as
Reduced Swap at c2:CD. The memory size of input derivative of c2:CD, $\Delta
D^{\prime}$ is $64\times 32\times 32\times 64\times$sizeof(float) and output
derivative $\Delta D$ has the same size. The memory size for weight $W$ is
$(64\times 3\times 3\times 64+64)\times$sizeof(float). Then, the theoretical
memory size, which needs to be kept allocated at c2:CD, is 32.14 MiB.
Considering the baseline (11 MiB), this is close to the peak memory
consumption with swapping. Thus, we can conjecture that NNTrainer’s swap
scheduling utilizes memory extremely efficiently.

There is almost the same memory reduction achieved for both On-Demand and
Reduced Swap; however, Reduced Swap reduces the latency by 10% with reduced
number of swapping by the EO-based memory planning proposed in §4. Proactive
Swap consumes more memory while reducing latency by 10%; it further reduces
swapping as shown in Figure 8.b and Figure 12. Note that Proactive Swap does
not request swapping at a convolution layer during forwarding (a red box in
Figure 12). It is because its activation layer incurs in-place computation,
which does not need additional memory swap-in. However, we need a swap-in
after the activation before its next layer; thus, the swapping count
increases. The latency of c9_layer:CD (green boxes with red borders) in Figure
13 compares the latency of the swap schemes.

###  5.3. Applications

Figure 14. Memory consumption of training applications.

Figure 14 shows the peak memory consumption of training with the batch size of
64, trained by NNTrainer (w/o Swap, Reduced Swap), TensorFlow, TensorFlow-Lite
(Python, C++), and PyTorch. The models are trained from scratch: AlexNet
(Krizhevsky et al., 2017), VGG16 (Simonyan and Zisserman, 2014), and Resnet18
(He et al., 2016). In every case, NNTrainer consumes much less memory. Without
swap, it consumes 23% (AlexNet), 15% (VGG16), and 35% (ResNet18) of what the
other uses. With Reduced Swap, it consumes 18% (AlexNet) and 5% (VGG16 and
ResNet18) of what the other uses; the most efficient of others, PyTorch,
consumes 20 times more! AlexNet shows less improvement with swapping is
because the last two linear layers consume peak memory; the weight dimension
is $4096\times 4096$, which is 62 MiB. The theoretical peak memory of Apply
Gradient is two times of its weight size due to the gradient (124 MiB).
Tensorflow-Lite, with both C++ and Python, consumes almost x46 times more
memory for ResNet18 even if it is for embedded devices.

Figure 15. CPU throttling of Tacotron2 (Galaxy S21 Ultra)

Tacotron2 (Shen et al., 2018) is a sequence-to-sequence text-to-speech model
based on attention, which enables end-to-end training with dataset of sentence
and voice. Developers can create a personalized text-to-speech (TTS)
application by running Tacotron2 on NNTrainer so that an application may read
books for kids with voices of their parents or translate with the user’s
voice. In our application, a user reads 18 sentences to create a user dataset
for personalization, which is combined with the pre-installed dataset.
Tacotron2 consists of an Encoder, Attention, and Decoder, and the Decoder
generates voice outputs; thus, we apply on-device training to the Decoder, and
other parts are kept static for inferences.

NNTrainer does time-unrolling to perform complex time iteration, and the
entire memory is statically allocated based on the maximum time iteration
declared by developers. Weights of the same layers that are time-unrolled
incur no additional memory or computation because of the Tensor sharing.
Because of time iteration, forward process is performed for all unrolled
layers, and gradients are accumulated in the backward process without updating
weights. The optimizer updates weights only once per layer, which requires
additional memory to store the gradient during time iterations. Gradient
Clipping (Pascanu et al., 2013) and Teacher Forcing (Williams and Zipser,
1989) are also supported.

In Figure 15, peak memory consumption and latency with the different numbers
of threads for training with 26 samples on Galaxy S21 Ultra are measured to
evaluate the effect of parallelism with OpenBLAS (Zhang Xianyi, 2013).
Overheating is a major issue of mobile phones, and to mitigate it, CPU cores
are usually throttled by limiting their frequencies. Training neural networks
heavily utilize CPUs; thus, we observe how the CPU throttling affects the
latency. Figure 15 shows the latency and frequency of each core, and we can
see that the CPU frequencies are continuously throttled down. Another finding
is that multi-core parallelism is not helpful for this model, and if we can
cool CPUs properly, the training performance will be improved significantly.
Only 364 MiB of memory is requested for the case without swap to personalize
the Tacotron2 and NNTrainer completes within ten minutes even with throttled
CPU cores, which has been enough to pass the quality assurance team.

Figure 16. Peak Memory Consumption of Transformer

Transformer (Vaswani et al., 2017) is a popular neural network model
throughout various applications. Transformer consists of Encoder and Decoder,
and each of them has stacked multi-head attention layers, which require a huge
amount of memory: over 6 GiB for 128 batch size, 6 stacked 8 multi-headed
attention layers in both encoder and decoder to train. Although it requires
memory-intensive computation, each multi-head attention layer is independent,
promoting easier and more efficient parallelism. Conventional frameworks
intentionally allocate weights of $Q,K,V$ for each head and try to achieve
shorter latency with parallelism. It is definitely efficient in clouds, which
have abundant resources; however, it is usually not desirable on devices
because the size of the internal activation of the attention layer increases.
For swapping, the smaller the size of the layer intermediate activation or
weight, the more reduction in the memory consumption. Therefore, NNTrainer
implements the multi-head attention using
$Concat(Attention(QV_{i}^{Q},KW_{i}^{K},VW_{i}^{V}))_{h}$ as proposed in
(Vaswani et al., 2017). Figure 16 shows the size of memory consumed to train
Transformer with different batch sizes. With relatively small batch sizes, the
effect of memory planner based on EOs is noticeable compared to PyTorch; it is
almost half (575.2 vs 256.9 MiB). However, as the batch size increases, the
size of intermediate activation saved for gradient computing grows
exponentially and the effect of the memory planer diminishes. Proactive Swap
reduces the memory consumption dramatically from 4.4 GiB to 0.22 GiB (down to
1/20) for 64 batch size. Moreover, only Proactive Swap can train with 128
batch size in a machine with 8 GiB RAM, utilizing only 420 MiB.

##  6\. Future Works

Our future plan is to make NNTrianer more widely available and expand for
wider range of machine learning techniques popular in embedded device
applications. Also, we do have a plan to develop new techniques to utilize
resources more efficiently.

  * •

Extend the reach of NNTrainer. This includes providing an interface with other
frameworks, where model importers will be our first priority followed by model
exporters, and expand platform support for Android with Java APIs, support
Yocto/OpenEmbedded and TIZEN with C# and Web APIs.

  * •

Extend support to various computation backends and accelerators for embedded
devices such as GPUs, DSPs and NPUs to expand computational capacity with
higher energy efficiency. Integer-based training or quantization-aware
training (Jacob et al., 2018) is the next step.

  * •

Support advanced few-shot learning techniques such as meta-learning and
adaption problems.

  * •

Support half precision during training. This is more efficient on-device at a
point of latency and memory consumption.

##  7\. Conclusions

We propose a highly efficient and light-weight on-device neural network
training framework, NNTrainer, with techniques based on novel observations on
neural network training mechanisms. We significantly reduce memory consumption
so that embedded devices can practically train neural networks without
deterioration of accuracy or modification of model architecture. The
evaluation results show the efficiency of NNTrainer for various neural network
models and applications. The proposed framework, NNTrainer, is practical open-
source software that personalizes complex AI services on mass-produced
devices.

## References

  * (1)
  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016.  TensorFlow: A System for Large-Scale Machine Learning. In _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)_. USENIX Association, Savannah, GA, 265–283.  https://www.usenix.org/conference/osdi16/technical-sessions/presentation/abadi
  * Allen (1971) David M Allen. 1971.  Mean square error of prediction as a criterion for selecting variables.  _Technometrics_ 13, 3 (1971), 469–475.
  * Baydin et al. (2018) Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2018.  Automatic differentiation in machine learning: a survey.  _Journal of machine learning research_ 18 (2018).
  * Bulatov (2018) Yaroslav Bulatov. 2018.  Fitting larger networks into memory.  https://medium.com/tensorflow/fitting-larger-networks-into-memory-583e3c758ff9.
  * Cai et al. (2020) Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. 2020.  Tinytl: Reduce memory, not parameters for efficient on-device learning.  _arXiv preprint arXiv:2007.11622_ (2020).
  * Chen et al. (2016) Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016.  Training deep nets with sublinear memory cost.  _arXiv preprint arXiv:1604.06174_ (2016).
  * Dahl et al. (2011) George E Dahl, Dong Yu, Li Deng, and Alex Acero. 2011.  Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition.  _IEEE Transactions on audio, speech, and language processing_ 20, 1 (2011), 30–42.
  * Delange et al. (2021) Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. 2021.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2021).
  * Dukhan (2019) Marat Dukhan. 2019.  The indirect convolution algorithm.  _arXiv preprint arXiv:1907.02129_ (2019).
  * Essalmi et al. (2010) Fathi Essalmi, Leila Jemni Ben Ayed, Mohamed Jemni, Sabine Graf, et al. 2010\.  A fully personalization strategy of E-learning scenarios.  _Computers in Human Behavior_ 26, 4 (2010), 581–591.
  * Google (2018) Google. 2018.  TensorFlow-Lite.  https://www.tensorflow.org/lite.
  * Ham et al. (2021) MyungJoo Ham, Jijoong Moon, Geunsik Lim, Jaeyun Jung, Hyoungjoo Ahn, Wook Song, Sangjung Woo, Parichay Kapoor, Dongju Chae, Gichan Jang, Yongjoo Ahn, and Jihoon Lee. 2021.  NNStreamer: Efficient and Agile Development of On-Device AI Systems. In _2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_. IEEE, 198–207.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hinton et al. (2012) Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. 2012\.  Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups.  _IEEE Signal processing magazine_ 29, 6 (2012), 82–97.
  * Huang et al. (2019) Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019\.  Gpipe: Efficient training of giant neural networks using pipeline parallelism.  _Advances in neural information processing systems_ 32 (2019).
  * Jacob et al. (2018) Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018.  Quantization and training of neural networks for efficient integer-arithmetic-only inference. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2704–2713.
  * Jia et al. (2014) Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014.  Caffe: Convolutional architecture for fast feature embedding. In _Proceedings of the 22nd ACM international conference on Multimedia_. 675–678.
  * Justin Basilico (2021) Justin Basilico. 2021.  Netflix Explains Recommendations and Personalization.  http:https://scale.com/blog/Netflix-Recommendation-Personalization-TransformX-Scale-AI-Insights.
  * Ketkar (2017) Nikhil Ketkar. 2017.  Stochastic gradient descent.  In _Deep learning with Python_. Springer, 113–132.
  * Krizhevsky et al. (2017) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2017.  Imagenet classification with deep convolutional neural networks.  _Commun. ACM_ 60, 6 (2017), 84–90.
  * Li et al. (2020) Li Li, Yuxi Fan, Mike Tse, and Kuo-Yi Lin. 2020.  A review of applications in federated learning.  _Computers & Industrial Engineering_ (2020), 106854.
  * Lim et al. (2021) Geunsik Lim, MyungJoo Ham, Jijoong Moon, and Wook Song. 2021.  LightSys: lightweight and efficient CI system for improving integration speed of software. In _2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_. IEEE, 1–10.
  * Lim et al. (2023) Geunsik Lim, Donghyun Kang, MyungJoo Ham, and Young Ik Eom. 2023.  SWAM: Revisiting Swap and OOMK for Improving Application Responsiveness on Mobile Devices. In _MobiCom 2023 (Annual International Conference On Mobile Computing And Networking), To appear_.
  * Ling et al. (2015) Zhen-Hua Ling, Shi-Yin Kang, Heiga Zen, Andrew Senior, Mike Schuster, Xiao-Jun Qian, Helen M Meng, and Li Deng. 2015.  Deep learning for acoustic modeling in parametric speech generation: A systematic review of existing techniques and future trends.  _IEEE Signal Processing Magazine_ 32, 3 (2015), 35–52.
  * Lu et al. (2015) Jie Lu, Vahid Behbood, Peng Hao, Hua Zuo, Shan Xue, and Guangquan Zhang. 2015.  Transfer learning using computational intelligence: A survey.  _Knowledge-Based Systems_ 80 (2015), 14–23.
  * MIT Technical Review ([n. d.]) MIT Technical Review. [n. d.].  On-Device AI.  https://www.technologyreview.com/hub/ubiquitous-on-device-ai/.  (accessed 14 Dec 2021).
  * Mostafa and Wang (2019) Hesham Mostafa and Xin Wang. 2019.  Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In _International Conference on Machine Learning_. PMLR, 4646–4655.
  * Narang et al. (2018) Sharan Narang, Gregory Diamos, Erich Elsen, Paulius Micikevicius, Jonah Alben, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2018\.  Mixed precision training. In _Proc. 6th Int. Conf. on Learning Representations (ICLR)_.
  * ndevilla (2017) ndevilla. 2017.  Iniparser4.  https://github.com/ndevilla/iniparser.
  * Pal and Pal (1993) Nikhil R Pal and Sankar K Pal. 1993.  A review on image segmentation techniques.  _Pattern recognition_ 26, 9 (1993), 1277–1294.
  * Pascanu et al. (2013) Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013.  On the difficulty of training recurrent neural networks. In _International conference on machine learning_. PMLR, 1310–1318.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017.  Automatic differentiation in pytorch. In _NIPS 2017 Workshop on Autodiff_ (Long Beach, California, USA).  https://openreview.net/forum?id=BJJsrmfCZ
  * Qian et al. (2014) Yao Qian, Yuchen Fan, Wenping Hu, and Frank K Soong. 2014.  On the training aspects of deep neural network (DNN) for parametric TTS synthesis. In _2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 3829–3833.
  * Rajbhandari et al. (2021) Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021.  Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_. 1–14.
  * Ren et al. (2021) Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021.  $\\{$ZeRO-Offload$\\}$: Democratizing $\\{$Billion-Scale$\\}$ Model Training. In _2021 USENIX Annual Technical Conference (USENIX ATC 21)_. 551–564.
  * Shah et al. (2020) Aashaka Shah, Chao-Yuan Wu, Jayashree Mohan, Vijay Chidambaram, and Philipp Krähenbühl. 2020.  Memory optimization for deep networks.  _arXiv preprint arXiv:2010.14501_ (2020).
  * Shen et al. (2018) Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. 2018\.  Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 4779–4783.
  * Shin et al. (2017) Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017.  Continual learning with deep generative replay.  _arXiv preprint arXiv:1705.08690_ (2017).
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Tan et al. (2018) Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. 2018.  A survey on deep transfer learning. In _International conference on artificial neural networks_. Springer, 270–279.
  * Teerapittayanon et al. (2016) Surat Teerapittayanon, Bradley McDanel, and H.T. Kung. 2016.  BranchyNet: Fast inference via early exiting from deep neural networks. In _2016 23rd International Conference on Pattern Recognition (ICPR)_. 2464–2469.  https://doi.org/10.1109/ICPR.2016.7900006
  * Thrun and Mitchell (1995) Sebastian Thrun and Tom M Mitchell. 1995.  Lifelong robot learning.  _Robotics and autonomous systems_ 15, 1-2 (1995), 25–46.
  * Torrey and Shavlik (2010) Lisa Torrey and Jude Shavlik. 2010.  Transfer learning.  In _Handbook of research on machine learning applications and trends: algorithms, methods, and techniques_. IGI global, 242–264.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Voigt and Von dem Bussche (2017) Paul Voigt and Axel Von dem Bussche. 2017.  The eu general data protection regulation (gdpr).  _A Practical Guide, 1st Ed., Cham: Springer International Publishing_ 10 (2017), 3152676.
  * Williams and Zipser (1989) Ronald J Williams and David Zipser. 1989.  A learning algorithm for continually running fully recurrent neural networks.  _Neural computation_ 1, 2 (1989), 270–280.
  * Wu et al. (2015) Ren Wu, Shengen Yan, Yi Shan, Qingqing Dang, and Gang Sun. 2015.  Deep image: Scaling up image recognition.  _arXiv preprint arXiv:1501.02876_ 7, 8 (2015).
  * Yang et al. (2019) Wenming Yang, Xuechen Zhang, Yapeng Tian, Wei Wang, Jing-Hao Xue, and Qingmin Liao. 2019.  Deep learning for single image super-resolution: A brief review.  _IEEE Transactions on Multimedia_ 21, 12 (2019), 3106–3121.
  * Zhang et al. (2020) Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. 2020.  Improved analysis of clipping algorithms for non-convex optimization.  _Advances in Neural Information Processing Systems_ 33 (2020), 15511–15521.
  * Zhang Xianyi (2013) Zhang Xianyi. 2013.  OpenBLAS.  http:https://github.com/xianyi/OpenBLAS.