###### Abstract

The union of Edge Computing (EC) and Artificial Intelligence (AI) has brought
forward the Edge AI concept to provide intelligent solutions close to the end-
user environment, for privacy preservation, low latency to real-time
performance, and resource optimization. Machine Learning (ML), as the most
advanced branch of AI in the past few years, has shown encouraging results and
applications in the edge environment. Nevertheless, edge-powered ML solutions
are more complex to realize due to the joint constraints from both edge
computing and AI domains, and the corresponding solutions are expected to be
efficient and adapted in technologies such as data processing, model
compression, distributed inference, and advanced learning paradigms for Edge
ML requirements. Despite the fact that a great deal of the attention garnered
by Edge ML is gained in both the academic and industrial communities, we
noticed the lack of a complete survey on existing Edge ML technologies to
provide a common understanding of this concept.To tackle this, this paper aims
at providing a comprehensive taxonomy and a systematic review of Edge ML
techniques, focusing on the soft computing aspects of existing paradigms and
techniques. We start by identifying the Edge ML requirements driven by the
joint constraints. We then extensively survey more than twenty paradigms and
techniques along with their representative work, covering two main parts: edge
inference, and edge learning. In particular, we analyze how each technique
fits into Edge ML by meeting a subset of the identified requirements. We also
summarize Edge ML frameworks and open issues to shed light on future
directions for Edge ML.

###### keywords:

edge artificial intelligence; edge machine learning; distributed learning;
distributed inference; federated learning; split learning; transfer learning;
model approximation; model compression

1 1 0 Academic Editor: Firstname Lastname 22 June 2023 6 August 2023 15 August
2023 https://doi.org/ A Comprehensive Review and a Taxonomy of Edge Machine
Learning: Requirements, Paradigms, and Techniques A Comprehensive Review and a
Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
Wenbin Li *, Hakim Hacid, Ebtesam Almazrouei and Merouane Debbah Wenbin Li,
Hakim Hacid, Ebtesam Almazrouei and Merouane Debbah Li, W.; Hacid, H.;
Almazrouei, E.; Debbah, M. Correspondence<EMAIL_ADDRESS>
##  1 Introduction

The tremendous success of Artificial Intelligence (AI) technologies Zhang et
al. (2022) in the past few years has been driving both industrial and societal
transformations through domains such as Computer Vision (CV), Natural Language
Processing (NLP), Robotics, Industry 4.0, Smart Cities, etc. This success is
mainly brought by deep learning, providing the conventional Machine Learning
(ML) techniques with the capabilities of processing raw data and discovering
intricate structures Lecun et al. (2015). Daily human activities are now
immersed with AI-enabled applications from content search and service
recommendation to automatic identification and knowledge discovery.

The existing ML models, especially deep learning models, such as GPT-4 OpenAI
(2023), Segment Anything Kirillov et al. (2023), OVSeg Liang et al. (2022),
Make-A-Video Singer et al. (2022), and Stable Diffusion Sta (2023), tend to
rely on complex model structures and large model size to provide competitive
performances. For instance, the largest WuDao 2.0 model Romero (2021) trained
on 4.9TB of data has surpassed state-of-the-art levels on nine benchmark tasks
with a striking 1.75 trillion parameters. As a matter of fact, large models
have clear advantages on multi-modality, multi-task, and benchmark
performance. However, such models require a relatively very large training
datasets to be built as well as a large amount of computing resources during
the training and inference phases. This dependency makes them usually closed
to public access, and unsuitable to be directly deployed for end devices or
even the small/medium enterprise level to provide real-time, offline, or
privacy-oriented services.

In parallel with ML development, Edge Computing (EC) was firstly proposed in
1990 Dilley et al. (2002). The main principle behind EC is to bring the
computational resources at locations closer to end-users. This was intended to
deliver cached content, such as images and videos, that are usually
communication-expensive, and prevent heavy interactions with the main servers.
This idea has later evolved to host applications on edge computing resources
Davis et al. (2004). The recent and rapid proliferation of connected devices
and intelligent systems has been further pushing EC from the traditional base
station level or the gateway level to the end device level. This offers
numerous technical advantages such as low latency, mobility, and location
awareness support to delay-sensitive applications Khan et al. (2019). This
serves as a critical enabler for emerging technologies like 6G, extended
reality, and vehicle-to-vehicle communications, to mention only a few.

Edge ML Lee et al. (2018), as the ML instantiation powered by EC and a union
of ML and EC, has brought the processing in ML to the network edge and adapted
ML technologies to the edge environment. In this work, the edge environment
refers to the end-user side pervasive environment composed of devices from
both the base station level and the end device level. In classical ML
scenarios, users run ML applications on their resource-constrained devices
(e.g., mobile phones, and Internet of Things (IoT) sensors and actuators),
while the core service is performed on the cloud server. In Edge ML, either
optimized models and services are deployed and executed in the end-user’s
device, or the ML models are directly built on the edge side. This computing
paradigm provides ML applications with advantages such as real-time immediacy,
low latency, offline capability, enhanced security and privacy, etc.

In order to illustrate the transformative potential and versatility of Edge ML
across different sectors, we briefly explore several application sectors where
Edge ML has already demonstrated substantial impact. The applications
represent just a fraction of the potential of Edge ML, while the breadth and
depth of Edge ML applications are expanding with the technique’s evolution and
its increased adoption.

  * •

Healthcare:  In healthcare, Edge ML enables real-time patient monitoring and
personalized treatment strategies. Wearable sensors and smart implants
equipped with Edge ML can process data locally, providing immediate health
feedback Amin and Hossain (2021). This advancement permits the early detection
of health irregularities and swift responses to potential emergencies, while
also maintaining patient data privacy by avoiding the need for data
transmission for analysis. Furthermore, in telemedicine, Edge ML could be used
to interpret diagnostic imaging locally and provide immediate feedback to
remote healthcare professionals, improving patient care efficiency and
outcomes.

  * •

Autonomous Vehicles: Edge ML is a key enabler for the advancements in the
field of autonomous vehicles, which includes both Unmanned Aerial Vehicles
(UAVs) and self-driving cars. These vehicles are packed with a myriad of
sensors and cameras that generate enormous amounts of data per second Yang et
al. (2021). Processing this data in real-time is crucial for safe and
efficient operation, and sending all the data to a cloud server is impractical
due to latency and bandwidth constraints. Edge ML, with its capability to
process data at the edge, can help in reducing the latency, and enhancing
real-time responses.

  * •

Smart City: Edge ML plays a crucial role in the realization of smart cities Lv
et al. (2021), where real-time data processing is paramount. Applications such
as intelligent traffic light control, waste management, and urban planning
greatly benefit from ML models that can analyze sensor data on-site and
respond promptly to changes in the urban environment. Moreover, Edge ML can
power public safety applications such as real-time surveillance systems for
crime detection and prevention. Here, edge devices like surveillance cameras
equipped with ML algorithms can detect unusual activities or behaviours and
alert relevant authorities in real time, potentially preventing incidents and
enhancing overall city safety.

  * •

Industrial IoT (IIoT): In the realm of Industrial IoT Tang et al. (2022), Edge
ML is instrumental in predictive maintenance and resource management. With ML
models running at the edge, real-time anomaly detection can be carried out to
anticipate equipment failures and proactively schedule maintenance.
Additionally, Edge ML can optimize operational efficiency by monitoring
production line performance, tracking resource usage, and automating quality
control processes.

However, the Edge ML’s core research challenge remains how to adapt ML
technologies to edge environmental constraints such as limited computation and
communication resources, unreliable network connections, data sensitivity,
etc., while keeping similar or acceptable performance. Research work was
carried out in the past few years tackling different aspects of this meta-
challenge, such as: model compression Cheng et al. (2017), transfer learning
Zhuang et al. (2021), and federated learning Abreha et al. (2022).

With the above-mentioned promising results in diverse areas, we noticed that
very little work has been realized to deliver a systematic view of relevant
Edge ML techniques, rather focusing on Edge ML in specific contexts. One
example worth reporting is Wang et al. Wang et al. (2020a, b), who present a
comprehensive survey on the convergence of edge computing and deep learning,
which covers aspects of hardware, communication, models, as well as edge
applications and edge optimization. The work is a good reference as an Edge ML
technology stack. On the other hand, the analysis of edge ML paradigms are
rather brief without a comprehensive analysis of diverse related problems and
the matching solutions. Abbas et al. Abbas et al. (2022) review the role and
impact of the relevant ML techniques in addressing the safety, security, and
privacy challenges in the specific context of the IoT systems. Mustafa et al.
Mustafa et al. (2022) center around two significant themes in edge computing:
Wireless Power Transfer (WPT) and Mobile Edge Computing (MEC). Their work
surveys the methodologies of offloading tasks in MEC and WPT to end devices,
and analyzes how the conjunction of WPT and MEC offloading can help overcome
limitations in smart device battery lifetime and task execution delay. Murshed
et al. Sarwar Murshed et al. (2021) introduce a machine learning survey at the
network edge, for which the training, inference, and deployment aspects are
briefly summarized, and the technique coverage is limited and only focuses on
representative techniques such as federated learning and quantization.

Compared to the existing works that briefly review the representative
techniques, our paper aims to provide a panoramic view of Edge ML
requirements, and offers a comprehensive technique review for edge machine
learning on the soft computing aspects of model training and model inference.
Throughout our review process, we strive for comprehensiveness, including more
than twenty technique categories and over fifty techniques in this paper. Our
work fills a significant gap in the literature by providing a single point of
reference that offers extensive coverage of the Edge ML field. The paper
delivers a more complete picture of the landscape of Edge ML, allowing readers
to understand the full breadth and depth of the available techniques, their
respective advantages and limitations, and their fit within different Edge ML
contexts.

In contrast with Mustafa et al. (2022), our paper concentrates on the
integration of ML techniques in the edge environment, dealing with the aspects
of data processing, model compression, distributed inference, and advanced
learning paradigms to explore a broader range of techniques and paradigms. In
comparison with Sarwar Murshed et al. (2021), which covers representative
works in topics of training, inference, applications, frameworks, software,
and hardware, our paper focuses on model training and inference computing
aspects, including a far richer list of techniques (e.g., in the edge learning
section, we include thirteen technique categories benefiting Edge ML, compared
to the three training techniques presented in Sarwar Murshed et al. (2021)).
We also analyze in detail the Edge ML requirements to provide a broad taxonomy
and show how each technique can satisfy different Edge ML requirements as a
systematic review.

Specifically, our paper answers the three following questions:

  * •

What are the computational and environmental constraints and requirements for
ML on the edge?

  * •

What are the Edge ML techniques to train intelligent models or enable model
inference while meeting Edge ML requirements?

  * •

How can existing ML techniques fit into an edge environment regarding these
requirements?

To answer the three above questions, this review is realized by firstly
identifying the Edge ML requirements, and then individually reviewing existing
ML techniques and analyzing if and how each technique can fit into edge by
fulfilling a subset of the requirements. Following this methodology, our goal
is to be as exhaustive as possible in the work coverage and provide a
panoramic view of all relevant Edge ML techniques with a special focus on
machine learning for model training and inference at the edge. Other topics,
such as Edge ML hardware Li and Liewig (2020) and edge communication Wang et
al. (2017), are beyond the scope of this paper. As such, we do not discuss
them in this review.

The remainder of the paper is organized as follows: Section 2 introduces the
Edge ML motivation driven by the requirements. Section 3 provides an overview
of all the surveyed edge ML techniques. In Sections 4 and 5, we describe each
technique and analyze them, respectively, in relation to Edge ML requirements.
Section 6 summarizes the technique review part, and Section 7 briefly
introduces the frameworks supporting Edge ML implementation. Section 8
identifies the open issues and future directions in Edge ML. Section 9
concludes our work and sheds light on future perspectives.

##  2 Edge Machine Learning: Requirements

In the context of machine learning, be it supervised learning, unsupervised
learning, or a reinforcement learning, an ML task could be either a training
or an inference. As in every technology, it is critical to understand the
underlying requirements that ensure proper expectations. By definition, the
edge infrastructure is generally resource-constrained in terms of the
following: computation power, i.e., processor and memory; storage capacity,
i.e., auxiliary storage; and communication capability, i.e., network
bandwidth. ML models, on the other hand, are commonly known to be hardware-
demanding, with computationally expensive and memory-intensive features.
Consequently, the union of EC and ML exhibits both constraints from edge
environment and ML models. When designing edge-powered ML solutions,
requirements from both the hosting environment and the ML solution itself need
to be considered and fulfilled for suitable, effective, and efficient results.

We introduce in this section the Edge ML requirements, structured in three
categories: (i) ML requirements, (ii) EC requirements, and (iii) overall
requirements, which are composite indicators from ML and EC for Edge ML
performance. The three categories of requirements are summarized in Figure 1.

###  2.1 ML Requirements

We foresee five main requirements an ML system should consider: (i) Low Task
Latency, (ii) High Performance, (iii) Generalization and Adaptation, (iv)
Labelled Data Independence, and (v) Enhanced Privacy and Security. We detail
these in the following.

  * •

Low Task Latency: Task latency refers to the end-to-end processing time for
one ML task, in seconds (s), and is determined by both ML models and the
supporting computation infrastructure. Low task latency is important to
achieve fast or real-time ML capabilities, especially for time-critical use-
cases such as autonomous driving. We use the term task latency instead of
latency to differentiate this concept from communication latency, which
describes the time for sending a request and receiving an answer.

  * •

High Performance: The performance of an ML task is represented by its results
and measured by general performance metrics such as top-n accuracy, and
f1-score in percentage points (pp), as well as use-case-dependent benchmarks
such as General Language Understanding Evaluation (GLUE) benchmark for NLP
Wang et al. (2018) or Behavior Suite for reinforcement learning Osband et al.
(2020).

  * •

Generalization and Adaptation: The models are expected to learn the
generalized representation of data instead of the task labels, so as to be
easily generalized to a domain instead of specific tasks. This brings the
models’ capability to solve new and unseen tasks and realize a general ML
directly or with a brief adaptation process. Furthermore, facing the disparity
between learning and prediction environments, ML models can be quickly adapted
to specific environments to solve the environmental specific problems.

  * •

Enhanced Privacy and Security: The data acquired from edge carry much private
information, such as personal identity, health status, and messages,
preventing these data from being shared in a large extent. In the meantime,
frequent data transmission over a network threatens data security as well. The
enhanced privacy and security requires the corresponding solution to process
data locally and minimize the shared information.

  * •

Labelled Data Independence: The widely applied supervised learning in modern
machine learning paradigms requires large amounts of data to train models and
generalize knowledge for later inference. However, in practical scenarios, we
cannot assume that all data in the edge are correctly labeled. The
independence of labelled data indicates the capability of an Edge ML solution
to solve one ML task without labelled data or with few labelled data.

Figure 1: Edge ML Requirements.

###  2.2 EC Requirements

Three main edge environmental requirements of EC impact the overall Edge ML
technology: (i) Computational Efficiency, (ii) Optimized Bandwidth, and (iii)
Offline Capability, summarized below.

  * •

Computational Efficiency: Refers to the efficient usage of computational
resources to complete an ML task. This includes both processing resources
measured by the number of arithmetic operations (OPs), and the required memory
measured in MB.

  * •

Optimized Bandwidth: Refers to the optimization of the amount of data
transferred over network per task, measured by MB/Task. Frequent and large
data exchanges over a network can raise communication and task latency. An
optimized bandwidth usage expects Edge ML solutions to balance the data
transfer over the network and local data processing.

  * •

Offline Capability: The edge connectivity of edge devices is often weak and/or
unstable, requiring operations to be performed on the edge directly. The
offline capability refers to the ability to solve an ML task when network
connections are lost or without a network connection.

###  2.3 Overall Requirements

The global requirements are composite indicators from ML and environmental
requirements for Edge ML performance. We specify four overall requirements in
this category: (i) Availability, (ii) Reliability, (iii) Energy Efficiency,
and (iv) Cost Optimization.

  * •

Availability: Refers to the percentage of time (in percentage points (pp))
that an Edge ML solution is operational and available for processing tasks
without failure. For edge ML applications, availability is paramount because
these applications often operate in real-time or near-real-time environments,
and downtime can result in severe operational and productivity loss.

  * •

Reliability: Refers to the ability of a system or component to perform its
required functions under stated conditions for a specified period of time.
Reliability can be measured using various metrics such as Mean Time Between
Failures (MTBF) and Failure Rate.

  * •

Energy Efficiency: Energy efficiency refers to the number of ML tasks obtained
per power unit, in Task/J. The energy efficiency is determined by both the
computation and communication design of Edge ML solutions and their supporting
hardware.

  * •

Cost optimization: Similar to energy consumption, edge devices are generally
low-cost compared to cloud servers. The cost here refers to the total cost of
realizing one ML task in an edge environment. This is again determined by both
the Edge ML software implementation and its supporting infrastructure usage.

It should be noted that, depending on the nature of Edge ML applications, one
Edge ML solution does not necessarily fulfill all the requirements above. The
exact requirements for each specific Edge ML application vary according to
each requirement’s critical level to an application. For example, for
autonomous driving, the task latency requirement is much more critical than
the power consumption and cost optimization requirements.

##  3 Techniques Overview

Figure 2 shows a global view of edge Machine Learning techniques reviewed in
this paper. We structure the related techniques into: (i) edge inference, and
(ii) edge learning. The edge inference category introduces the technologies to
accelerate the task latency of ML model inference. This is performed through,
e.g., compressing existing models to consume less hardware resources or by
dividing existing models into several parts for parallel inference
collaboration. The edge learning category introduces solutions to directly
build ML models on the edge side by learning locally from edge data. We detail
the categories in the next sections.

Figure 2: Edge ML Technique Overview.

Before introducing the details of each reviewed technique, we go through three
basic machine learning paradigms, i.e., supervised learning, unsupervised
learning, and reinforcement learning, to lay the theoretical foundation of ML.
Briefly, supervised learning involves using an ML model to learn a mapping
function between input data and the target variable from the labeled dataset.
Unsupervised learning directly describes or extracts relationships in
unlabeled data without any guidance from labelled data. Reinforcement learning
is the process that an ML agent continuously interacts with its environment,
performs actions to obtain awards, and learns to achieve a goal by the trial-
and-error method.

Extending the work from Huisman et al. (2021), we give below the formal
definition of the three basic learning paradigms. The objective of the
definition is not merely to offer a conventional understanding of these
methods, but more importantly, to create a conceptual bridge between these
mainstream learning techniques and their specialized applications and
adaptations for edge learning techniques, which helps to understand how these
well-established AI techniques transform when applied to the context of Edge
computing. Breakthroughs have been made in all three ML learning paradigms to
derive meaningful data insights and bring intelligent capabilities, and the
reviewed techniques in this paper all fit into the three general machine
learning paradigms.

###  3.1 Supervised Learning

Supervised learning learns a function $f_{\theta}:X\rightarrow Y$ mapping
inputs $x_{i}\in X$ to the corresponding outputs $y_{i}\in Y$ with the help of
a labeled dataset $D$ of $m$ samples $D=\\{(x_{i},y_{i})\\}_{i=1}^{m}$, in
which $\theta$ are ML model parameters (e.g., weights and biases in the case
of neural network). The learning process aims at finding optimal or sub-
optimal values for $\theta$ specific to the dataset $D$ that minimizes an
empirical loss function $L_{D}$ through a training process (e.g., backward
propagation in the case of neural network) as:

$\theta_{SL}:=\underset{\theta}{arg\,min}\ L_{D}(\theta),$ (1)

where $SL$ stands for “supervised learning”. In practice, the labelled dataset
$D$ is often divided into training, validation, and testing datasets $D^{tr},\
D^{val},\ D^{test}$ to train the model, guide the training process, and
evaluate model performance after training, respectively Xu and Goodacre
(2018).

Finding globally optimal values of $\theta_{SL}$ is computationally expensive,
while in practice the training process is commonly an approximation to find
sub-optimal $\theta_{SL}$ values guided by a predefined meta-knowledge
$\omega$ including the initial model parameters $\theta$, the training
optimizer, and learning rate in the case of neural network, as:

$\theta_{SL}\approx g_{\omega}(D,L_{D}),$ (2)

where $g_{\omega}$ is an optimization procedure that uses predefined meta-
knowledge $\omega$, dataset $D$ and loss function $L_{D}$ to continuously
update the model’s parameters $\theta$ and output final $\theta_{SL}$.

###  3.2 Unsupervised Learning

Training an ML model in the unsupervised manner is mostly similar to the
supervised learning processing, except that the learned function
$f_{\theta}:X\rightarrow X$ mapping input $x_{i}\in X$ to the same input
$x_{i}$ or other inputs. Unsupervised learning only uses unlabeled dataset
$\bar{D}$ of $n$ sample $\bar{D}=\\{(x_{i})\\}_{i=1}^{n}$ to determine
$\theta$ values specific to the dataset $\bar{D}$ that minimize an empirical
loss function $L_{\bar{D}}$ through a training process, as:

$\theta_{UL}:=\underset{\theta}{arg\,min}\ L_{\bar{D}}(\theta),$ (3)

where $UL$ stands for “unsupervised learning”. Furthermore, the same
approximation is applied to unsupervised learning to efficiently fit the
$\theta_{UL}$ to $\bar{D}$:

$\theta_{UL}\approx g_{\omega}(\bar{D},L_{\bar{D}})$ (4)

In addition to the above unsupervised learning paradigm which is used to train
ML models, other unsupervised learning techniques such as clustering
Golalipour et al. (2021) apply predefined algorithms and computing steps to
directly generate expected outputs (e.g., data clusters) from $\bar{D}$. In
such context, the unsupervised learning approximates the values of specific
algorithms’ hyperparameters $\bar{\theta}_{UL}$ as:

$\bar{\theta}_{UL}\approx g_{\omega}(\bar{D},L_{\bar{D}})$ (5)

###  3.3 Reinforcement Learning

In the classic scenario of reinforcement learning where agents know the state
at any given time step, the reinforcement learning paradigm can be formalized
into a Markov Decision Process (MDP) as $M=(S,A,P,r,p_{0},\gamma,T)$, where
$S$ is the set of states, $A$ the set of actions, P the transition probability
distribution defining $P(s_{t+1}|s_{t},a_{t})$ the transition probability from
$s_{t}$ to $s_{t+1}$ via $a_{t}$, $r:S\times A\rightarrow\mathbb{R}$ the
reward function, $p_{0}$ the probability distribution over initial states,
$\gamma\in[0,1]$ the discount factor prioritizing short- or long-term rewards
by respectively decreasing or increasing it, $T$ the maximum number of time
steps. At a time step $t\in T$, a policy function $\pi_{\theta}$, usually
represented by a model in the case of deep reinforcement learning, is used to
determine the action $a_{t}$ that an agent performs at state
$s_{t}:a_{t}=\pi_{\theta}(s_{t})$, where $\theta$ is the parameters of the
policy function; after the action $a_{t}$, the agent receives an award
$r_{t}=r(s_{t},\pi_{\theta}(s_{t})),r_{t}\in\mathbb{R}$ and enters into a new
state $s_{t+1}$. The interaction between agent and environment stops until a
criterion is met, such as the rewards are maximized.

The objective of the reinforcement learning is to make agents learn to act and
maximize the received rewards as follows:

$\theta_{RL}:=\underset{\theta}{arg\,min}\
\mathbb{E}_{traj}\sum_{t=1}^{T}\gamma^{t}r(s_{t},\pi_{\theta}(s_{t})),$ (6)

where $RL$ stands for “reinforcement learning”, and $\mathbb{E}_{traj}$ is the
expectation over possible trajectories $traj=(s_{0},\pi_{\theta}(s_{0}),\
...,\ s_{T},\pi_{\theta}(s_{T}))$.

Similar to supervised and unsupervised learning, the sub-optimum of
$\theta_{RL}$ are searched via an approximation process:

$\theta_{RL}\approx g_{\omega}(M,L_{M}),$ (7)

where $g_{\omega}$ is an optimization procedure that uses predefined meta-
knowledge $\omega$, the given MDP $M$ and loss function $L_{M}$ to produce
final $\theta_{RL}$.

##  4 Edge Inference

Edge inference techniques seek to enable large model inference on edge devices
and accelerate the inference efficiency. The techniques can be categorized
into three main groups: (i) model compression and approximation, (ii)
distributed inference, and (iii) other inference acceleration techniques.

###  4.1 Model Compression and Approximation

A large amount of redundancy among the ML model parameters (e.g., neural
network weights) has been observed Denil et al. (2013), showing that a small
subset of the weights is sufficient to reconstruct the entire neural network.
Model compression and approximation are methods to transform ML models into
smaller-size or approximate models with low-complexity computations. This is
performed with the objective to reduce the memory use and the arithmetic
operations during the inference, while keeping acceptable performances. Model
compression and approximation can be broadly classified into three categories
Wang et al. (2019): (i) Quantization, (ii) Weight Reduction, and (iii)
Activation Function Approximation. We discuss these categories in the
following:

####  4.1.1 Quantization

Quantization is the process of converting ML model parameters $\theta$ (i.e.,
weights and bias in neural networks) and activation outputs, represented in
Floating Point (FP) format of high precision such as FP64 or FP32, into a low-
precision format and then perform computing tasks such as training or
inference. Different formats of quantization can be summarized as:

  * •

Low-Precision Floating-Point Representation: A floating-point parameter
describes binary numbers in the exponential form with an arbitrary binary
point position such as 32-bit floating point (FP32), 16-bit floating point
(FP16), 16-bit Brain Floating Point (BFP16) Wang and Kanwar (2021).

  * •

Fixed-Point Representation: A fixed-point parameter Goyal et al. (2021) uses
predetermined precision and binary point locations. Compared to a high-
precision floating-point representation, the fixed-point parameter
representation can offer faster, cheaper, and more power-efficient arithmetic
operations.

  * •

Binarization and Terrorization: Binarization Yuan and Agaian (2023) is the
quantization of parameters into just two values, typically -1, 1 with a
scaling factor. The terrorization Liu et al. (2023) on the other hand adds the
value 0 to the binary value set to express 0 in models.

  * •

Logarithmic Quantization: In a logarithmic quantization Lee et al. (2017),
parameters are quantized into powers of two with a scaling factor. Work in Lai
et al. (2017) shows that a weight’s representation range is more important
than its precision in preserving network accuracy. Thus, logarithmic
representations can cover wide ranges using fewer bits, compared to the other
above-mentioned linear quantization formats.

In addition to the main exploited data types, AI-specific data formats and
several quantization contributions exist in the literature and are introduced
in Table 1.

Table 1: AI-specific data formats and model quantization works.
---
Work |  Contribution |  Results and Insights
Posit Representation Gustafson and Yonemoto (2017) |  The Posit is a data type designed to supersede IEEE Standard 754 floating-point numbers 461 (2008). |  Provides larger dynamic range and higher accuracy than traditional floats, along with simpler hardware implementation and exception handling.
Fixed-Posit Representation Gohil et al. (2021) |  Extended the Posit data type by maintaining a fixed count of the bits, unlike conventional posits that have a configurable number of regime and exponent bits. |  Enhances the power and area efficiency compared to both Posit and 32-bit IEEE-754 floating-point representations. Improvements of up to 70% in power, 66% in area, and 26% in delay.
Tensor Cores NVIDIA Corporation (2022) |  NVIDIA’s Tensor Cores is specialized hardware designed for performing the tensor/matrix computations with mixed-precision computations, enabling FP8 in the Transformer Engine, Tensor Float 32 (TF32) tf (3), and FP16. |  Provide an order-of-magnitude higher performance with reduced precisions
8-bit Quantization Jacob et al. (2018) |  An 8-bit quantization schema for MobileNet Howard et al. (2017) on the ARM NEON-based implementation. |  Model size reduction: 4x. Inference task latency reduction: up to 50%. Accuracy drop: 1.8% for COCO datasett Lin et al. (2014).
SLQ Lee et al. (2019) |  A successive logarithmic quantization (SLQ) scheme to quantize the training error again when the quantization error is higher than a certain threshold. |  Accuracy drop: less than 1.5% for AlexNet Krizhevsky et al. (2017), SqueezeNet Iandola et al. (2016), and VGG-S Jin et al. (2021) at 4 to 5-bit weight representation.
SLQ Training Oh et al. (2021) |  A specific training method for the Successive Logarithmic Quantization (SLQ) scheme. |  Performance degradation of around 1% at 3-bit weight quantization.
Binary Network Qin et al. (2022) |  An accurate and efficient binary neural network for keyword-spotting applications, along with a binarization-aware training method for optimization. |  Impressive 22.3 times speedup of task latency and 15.5 times storage-saving with only less than 3% accuracy drop on Google Speech Commands V1-12 task Warden (2018).
Binary Distilled Transformer Liu et al. (2022) |  A binarized multi-distilled transformer including a two-set binarization scheme, an elastic binary activation function with learned parameters, and a method for successively distilling models. |  Accuracy of fully binarized transformer models approaches a full-precision BERT baseline on the GLUE language understanding benchmark within as little as 5.9%.
FMA Osorio et al. (2022) |  A new class of Fused Multiply–Add (FMA) operators built on BFP16 arithmetic while maintaining accuracy comparable to that of the standard FP32. |  Improved performance by 1.28–1.35$\times$ on ResNet compared to FP32.

To produce the corresponding quantized model, post-training quantization and
quantization-aware training can be applied. Given an existing trained model,
post-training quantization directly converts the trained model parameters
and/or activation according to the conversion needs, to reduce model size and
improve task latency during the inference phase. On the other hand, and
instead of quantizing existing models, quantization-aware training is a method
that trains an ML model by emulating inference-time quantization, which has
proved to be better for model accuracy Zhang et al. (2023). During the
training of a neural network, quantization-aware training simulates low-
precision behavior in the forward pass, while the backward pass based on
backward propagation remains the same. The training process takes into account
both errors from training data labels as well as quantization errors which
accumulate in the total loss of the model, and hence the optimizer tries to
reduce them by adjusting the parameters accordingly. Similarly, the work De
Putter and Corporaal (2022) analyzes the significant trade-off between energy
efficiency and model accuracy, showing that the application of repair methods
(e.g., ReAct-Net Liu et al. (2020)), could largely offset the accuracy loss
after quantization. Zhou et al. Zhou et al. (2016) analyzed various data
precision combinations, concluding that accuracy deteriorates rapidly when
weights are quantized to fewer than four bits. The work Ruospo et al. (2021)
investigates the impact of data representation and bit-width reduction on CNN
resilience, particularly in the context of safety-critical and resource-
constrained systems. The results indicate that fixed-point data representation
offers a superior trade-off between memory footprint reduction and resilience
to hardware faults, especially for the LeNet-5 network, achieving a 4x memory
footprint reduction at the expense of less than 0.45% critical faults without
requiring network retraining.

Overall, moving from high-precision floating-point to lower-precision data
representations is especially useful for ML models on edge devices with only
low precision operation support such as Application-Specific Integrated
Circuit (ASIC) and Field Programmable Gate Arrays (FPGA) to facilitate the
trade-off between task accuracy and task latency. Quantization reduces the
precision of parameters and/or activation, and thereby decreases the inference
task latency by reducing the consumption of computing resources, while the
workload reduction brought by cheaper arithmetic operations leads to energy
and cost optimization as well. Moreover, quantization techniques make it
feasible to run models on low-resource edge devices, increasing the
availability of the application by allowing it to function with low
connectivity. One the other hand, the techniques can also reduce the
application robustness or resilience to hardware faults. Reduced precision
leads to a model that is more susceptible to errors due to slight changes in
the input or due to hardware faults, and thus decreases the reliability.

####  4.1.2 Weight Reduction

Weight reduction is a class of methods that removes redundant parameters from
$\theta$ through pruning and parameter approximation. We reviewed the three
following categories of methods in this paper:

  * •

Pruning. The process of removing redundant or non-critical weights and/or
nodes from models Cheng et al. (2017): weight-based pruning removes
connections between nodes (e.g., neurons in neural network) by setting
relevant weights to zero to make the ML models sparse, while node-based
pruning removes all target nodes from the ML model to make the model smaller.

  * •

Weight Sharing. The process of grouping similar model parameters into buckets
and reuse shared weights in different parts of the model to reduce model size
or among models Chu et al. (2021) to facilitate the model structure design.

  * •

Low-rank Factorization. The process of decomposing the weight matrix into
several low-rank matrices by uncovering explicit latent structures Jaderberg
et al. (2014).

A node-based pruning method is introduced in Srinivas and Babu (2015) to
remove redundant neurons in trained CNNs. In this work, similar neurons are
grouped together following a similarity evaluation based on squared Euclidean
distances and then pruned away. Experiments showed that the pruning method can
remove up to 35% nodes in AlexNet with a 2.2% accuracy loss on the dataset of
ImageNet Russakovsky et al. (2015). A grow-and-prune paradigm is proposed in
Dai et al. (2019) to complement network pruning to learn both weights and
compact Deep neural networks(DNNs) architectures during training. The method
iteratively tunes the architecture with gradient-based growth and pruning of
neurons and weight. Experimental results showed the compression ratio of
15.7$\times$ and 30.2$\times$ for AlexNet and VGG-16 network, respectively.
This delivers a significant parameter and arithmetic operation reduction
relative to pruning-only methods. In practice, pruning is often combined with
a post-tuning or a retraining process to improve the model accuracy after
pruning Yu et al. (2017). A Dense–Sparse–Dense training method is presented in
Han et al. (2017), which introduces a post-training step to re-dense and
recover the original model symmetric structure to increase the model capacity.
This was shown to be efficient as it improves the classification accuracy by
1.1% to 4.3% on ResNet-50 He et al. (2016), ResNet-18 He et al. (2016), and
VGG-16 Simonyan and Zisserman (2015). The pruning method of SparseGPT is
proposed in Frantar and Alistarh (2023), showing that the large-scale
generative pretrained transformer (GPT) family models can be pruned to at
least 50% sparsity in one-shot, without any retraining, at minimal loss of
accuracy. The driving idea behind this is an approximate sparse regression
solver that runs entirely locally and relies solely on weight updates designed
to preserve the input–output relationship for each layer. We also examine the
layer-wise weight-pruning method presented in Wu et al. (2021). The method
relies on a differential evolutionary layer-wise weight pruning operating in
two distinct phases—a model-pruning phase, which analyzes each layer’s pruning
sensitivity and guides the pruning process, and a model-fine-tuning phase,
where removed connections are considered for recovery to improve the model’s
capacity. Notably, the approach achieved impressive compression ratios of at
least 10$\times$ across different models, with a standout 29$\times$
compression achieved for AlexNet. Another notable development in the field of
structural pruning is the Dependency Graph (DepGraph) method Fang et al.
(2023). This method represents a breakthrough in tackling the complex task of
any structural pruning across a broad variety of neural architectures, from
CNNs and RNNs to GNNs and Transformers. DepGraph introduces an automated
system that models the dependency between layers to effectively group
parameters that can be pruned together. The method demonstrates a promising
performance across a multitude of tasks and architectures, including ResNet,
DenseNet, MobileNet, and Vision transformer for images, GAT for graph, DGCNN
for 3D point cloud, alongside LSTM for language.

The aforementioned pruning methods are static, as they permanently change the
original network structure which may lead to a decrease in model capability.
On the other hand, dynamic pruning Liang et al. (2021) determines at run-time
which layers, image channels (for CNN), or neurons would not participate in
further model computing during a task. A dynamic channel pruning is proposed
in Gao et al. (2019). This method dynamically selects which channel to skip or
to process using feature boosting and suppression, which is achieved by the
use of a side network trained together along the CNN to guide channel
amplification and omission. This work achieved a 2$\times$ acceleration on
ResNet-18 with 2.54% top-1 and 1.46% top-5 accuracy loss, respectively.

A multi-scale weight-sharing method is introduced in Aich et al. (2020) to
share weights among the convolution kernels of the same layer. To share kernel
weights for multiple scales, the shared tuple of kernels is designed to have
the same shape, and different kernels in the shared tuple are applied to
different scales. With approximately 25% fewer parameters, the shared-weight
ResNet model provides similar performance compared to the baseline ResNets He
et al. (2016). Instead of looking up tables to locate the shared weight for
each connection, HashedNets is proposed in Chen et al. (2015) to randomly
group connection weights into hash buckets via a low-cost hash function. These
weights are tuned to adjust to the HashedNets weight sharing architecture with
standard back-propagation during the training. Evaluations showed that
HashedNets achieved a compression ratio of 64% with an around 0.7% accuracy
improvement against a five-layer CNN baseline with the MNIST dataset LeCun et
al. (1998). Furthermore, the recent work Li et al. (2022) uses a gradient-
based method to determine a threshold for attention scores at runtime, thereby
effectively pruning inconsequential computations without significantly
affecting model accuracy. Their proposed bit-serial architecture, known as
LeOPArd, leverages this gradient-based thresholding to enable early
termination, resulting in a significant boost in computational speed
(1.9$\times$ on average) and energy efficiency (3.9$\times$ on average), with
a minor trade-off in accuracy (<0.2% degradation). The related work of pruning
is summarized in Table 2.

Table 2: Summary of pruning works and main results.
---
Methods |  Target Models |  Main Results
Node-based Pruning Srinivas and Babu (2015) |  CNNs |  35% pruning, 2.2% accuracy loss
Grow-and-Prune Dai et al. (2019) |  DNNs |  15.7$\times$, 30.2$\times$ compression
Dense-Sparse-Dense Han et al. (2017) |  ResNet, VGG-16 |  1.1% to 4.3% accuracy increase
SparseGPT Frantar and Alistarh (2023) |  GPT models |  50% sparsity
Differential Layer-Wise Pruning Wu et al. (2021) |  LeNet, AlexNet, VGG16 |  10$\times$ to 29$\times$ compression
DepGraph Structural Pruning Fang et al. (2023) |  Any Model Structure |  8–16$\times$ speedup, $-$6% to +0.3% accuracy change
Dynamic Channel Pruning Gao et al. (2019) |  ResNet-18 |  2$\times$ acceleration, 2.54% top-1, 1.46% top-5 accuracy loss
Multi-Scale Weight Sharing Aich et al. (2020) |  ResNets |  25% fewer parameters
HashedNets Chen et al. (2015) |  5-layer CNN |  64% compression, 0.7% accuracy improvement
Gradient-Based Pruning Li et al. (2022) |  Transformers |  1.9$\times$ speed, 3.9$\times$ energy efficiency, $<$0.2% accuracy loss

Structured matrices use repeated patterns within matrices to represent model
weights to reduce the number of parameters. The circulant matrix, in which all
row vectors are composed of the same elements and each row vector is shifted
one element to the right relative to the preceding row vector, are often used
as the structured matrix to provide a good compression and accuracy for RNN-
type models Wang et al. (2017, 2018). The Efficient Neural Architecture Search
(Efficient NAS) via parameter sharing is proposed in Pham et al. (2018), in
which only one shared set of model parameters is trained for several model
architectures, also known as child models. The shared weights are used to
compute the validation losses of different architectures. Sharing parameters
among child models allows efficient NAS to deliver strong empirical
performances for neural network design and use fewer GPU FLOP than automatic
model design approaches. The NAS approach has been successfully applied to
design model architectures for different domains Liu et al. (2023) including
CV and NLP.

As for low-rank factorization, to find the optimal decomposed matrices to
substitute the original weight matrix, Denton et al. Denton et al. (2014)
analyze three decomposition methods on pre-trained weight matrices: (i)
singular-value decomposition, (ii) canonical polyadic decomposition, and (iii)
blustering approximation. Experimental results on a 15-layer CNN demonstrate
that singular-value decomposition achieved the best performance by a
compression ratio of 2.4$\times$ to 13.4$\times$ on different layers along
with a 0.84% point of top-one accuracy loss in the ImageNet dataset. A more
recent work Chen et al. (2021) proposes a data-aware low-rank compression
method (DRONE) for weight matrices of fully-connected and self-attention
layers in large-scale NLP models. As weight matrices in NLP models, such as
BERT Devlin et al. (2019), do not show obvious low-rank structures, a low-rank
computation could still exist when the input data distribution lies in a lower
intrinsic dimension. The proposed method considers both the data distribution
term and the weight matrices to provide a closed-form solution for the optimal
rank-k decomposition. Experimental results show that DRONE can achieve
1.92$\times$ speedup on the Microsoft Research Paraphrase Corpus (MRPC) Dolan
and Brockett (2005) task with only 1.5% loss in accuracy, and when DRONE is
combined with distillation, it reaches 12.3$\times$ speedup on natural
language inference tasks of MRPC, Corpus of Linguistic Acceptability (CoLA)
Warstadt et al. (2019), and Semantic Textual Similarity (STS) STSBenchmark
(2019).

Overall, weight reduction directly reduces the ML model size by removing
uncritical parameters. When performing tasks after weight reduction, ML models
use less memory and require fewer arithmetic operations, which directly reduce
the task latency with less workload and improve the computational resource
efficiency. This is critical for time-sensitive applications to improve the
perceived availability and responsiveness of the system. In addition, such
improvements contribute to optimized energy consumption and cost. Similar to
quantization, the weight-reduction techniques potentially make the model less
resilient to certain types of hardware faults and such decrease the
reliability. For example, a fault that affects a critical weight in a pruned
network might have a bigger impact on the output than the same fault in an
unpruned network, simply because there are fewer weights to ’absorb’ the
fault.

####  4.1.3 Knowledge Distillation

Knowledge Distillation is a procedure where a neural network is trained on the
output of another network along with the original targets in order to transfer
knowledge between the ML model architectures Borup and Andersen (2021). In
this process, a large and complex network, or an ensemble model, is trained
with a labelled dataset for a better task performance. Afterwards, a smaller
network is trained with the help of the cumbersome model via a loss function
$L$, measuring the output difference of the two models. This small network
should be able to produce comparable results, and in the case of over-fitting,
it can even be made capable of replicating the results of the cumbersome
network.

A knowledge-distillation framework for a fast objects detection task is
proposed in Chen et al. (2017). To address the specific challenges of object
detection in the form of regression, region proposals, and less-voluminous
labels, two aspects are considered: (i) a weighted cross-entropy loss, to
address the class imbalance, and (ii) a teacher-bounded loss, to handle the
regression component and adaptation layers to better learn from intermediate
teacher distributions. Evaluations with the datasets of Pattern Analysis,
Statistical Modelling and Computational Learning (PASCAL) Everingham et al.
(2010), Karlsruhe Institute of Technology and Toyota Technological Institute
(KITTI) Geiger et al. (2013), and COCO showed accuracy improvements by 3% to
5%. Wen et al. Wen et al. (2021) argued that overly uncertain supervision of
teachers can negatively influence the model’s results. This is due to the fact
that the knowledge from a teacher is useful but still not exactly right
compared with a ground truth. Knowledge adjustment and dynamic temperature
distillation are introduced in this work to penalize incorrect supervision and
overly uncertain predictions from the teacher, making student models more
discriminatory. Experiments on CIFAR-100 Krizhevsky and Hinton (2009),
CINIC-10 Darlow et al. (2018), and Tiny ImageNet Le and Yang (2015) showed
nearly state-of-the-art method accuracy.

MiniVit Zhang et al. (2022) proposes to compress vision transformers with
weight sharing across layers and weight distillation. A linear transformation
is added on each layer’s shared weights to increase weight diversity. Three
types of distillation for transformer blocks are considered in this work: (i)
prediction-logit distillation, (ii) self-attention distillation, and (iii)
hidden-state distillation. Experiments showed MiniViT can reduce the size of
the pre-trained Swin-B transformer by 48% while achieving an increase of 1.0%
in Top-1 accuracy on ImageNet.

Overall, knowledge distillation directly reduces the ML model size by
simplifying model structures. Compared to the source model, the target model
has a more compact and distilled structure with less parameters. Hence the
workload of a task is reduced, leading to a better computational efficiency,
higher availability, low task latency, and optimized energy consumption and
cost. The distilled models potentially have higher reliability because they
exert less stress on the hardware, reducing the likelihood of hardware faults
or overheating. However, depending on the specific setup, the faults could
have a greater impact on the model applications.

####  4.1.4 Activation Approximation

Besides the neural network’s size complexity, i.e., in terms of the number of
parameters, and architecture complexity, i.e., in the terms of layers,
activation functions also impact the task latency of a neural network.
Activation functions approximation replaces non-linear activation functions
(e.g., sigmoid  and tanh) in ML models with less computationally expensive
functions (e.g., ReLU) to simplify the calculation or convert the
computationally expensive calculation to series of lookup tables.

In an early work Amin et al. (1997), the Piece-wise Linear Approximation of
Non-linear Functions (PLAN) was studied. The sigmoid function was approximated
by a combination of straight lines, and the gradient of the lines were chosen
such that all the multiplications were replaced by simple shift operations.
Compared to sigmoid and tanh, Hu et al. Hu et al. (2021) show that ReLU, among
other linear functions, is not only less computationally expensive but also
proved to be more robust to handle the neural network vanishing gradient
problem, in which the error dramatically decreases along with the back-
propagation process in deep neural networks.

Activation approximation improves the computing resource usage by reducing the
required number of arithmetic operations in ML models, and thus decreases the
task latency with an acceptable increase in task error.

###  4.2 Distributed Inference

Distributed Inference divides ML models into different partitions and carries
out a collaborative inference by allocating partitions to be distributed over
edge resources and computing in a distributed manner Zhao et al. (2018).

The target edge resources to distribute the inference task can be broadly
divided into three levels: (i) local processors in the same edge device Lane
et al. (2016), (ii) interconnected edge devices Zhao et al. (2018), and (iii)
edge devices and cloud servers Li et al. (2018). Among the three levels, an
important research challenge is to identify the partition points of ML models
by measuring data exchanges between layers to balance the usage of local
computational resources and bandwidth among distributed resources.

To tackle the tightly coupled structure of CNN, a model parallelism
optimization is proposed in Du et al. (2021), where the objective is to
distribute the inference on edge devices via a decoupled CNN structure. The
partitions are optimized based on channel group to partition the convolutional
layers and then an input-based method to partition the fully connected layers,
further exposing the high degree of parallelism. Experiments show that the
decoupled structure can accelerate the inference of large-scale ResNet-50 by
3.21$\times$ and reduce 65.3% memory use with 1.29% accuracy improvement.
Another distributed inference framework is also proposed in Hemmat et al.
(2022) to decompose a complex neural network into small neural networks and
apply class-aware pruning on each small neural network on the edge device. The
inference is performed in parallel while considering the available resources
on each device. The evaluation shows that the framework achieves up to
17$\times$ speed-up when distributing a variant of VGG-16 over 20 edge
devices, with around a 0.5% loss in accuracy.

Distributed inference can improve the end-to-end task latency by increasing
the computing parallelism over a distributed architecture. At a price of
bandwidth usage and network dependency, the overall energy efficiency and cost
are optimized. By distributing the inference task, the load on individual
devices is reduced, allowing more tasks to be processed concurrently, which
can increase the availability of the ML application. In a distributed
configuration, if one node fails, the task can be reassigned to another node,
thereby increasing the overall system reliability.

###  4.3 Other Inference Acceleration Techniques

There exist other ways for accelerating inference in the literature. These
have been categorized in a separate category as they are not as popular as the
previously discussed techniques. These include: (i) Early Exit of Inference
(EEoI), (ii) Inference Cache, and (iii) Model-Specific Inference Acceleration.
We briefly review them in the following.

####  4.3.1 Early Exit of Inference (EEoI)

The Early Exit of Inference (EEoI) is powered by a deep network architecture
augmented with additional side branch classifiers Teerapittayanon et al.
(2016). This allows prediction results for a large portion of test samples to
exit the network early via these branches when samples can already be inferred
with high confidence.

BranchyNet, proposed in Teerapittayanon et al. (2016), is based on the
observation that features learned at an early layer of a network may often be
sufficient for the classification of many data points. By adding branch
structures and exit criteria to neural networks, BranchyNet is trained by
solving a joint optimization problem on the weighted sum of the loss functions
associated with the exit points. During the inference, BranchyNet uses the
entropy of a classification result as a measure of confidence in the
prediction at each exit point and allows the input sample to exit early if the
model is confident in the prediction. Evaluations have been conducted with
LeNet LeCun et al. (1998), AlexNet, and ResNet on MNIST, CIFAR-10 datasets,
showing BranchyNet can improve accuracy and significantly reduce the inference
time of the network by 2$\times$–6$\times$.

To improve the modularity of the EEoI methods, a plug-and-play technique named
Patience-based Early Exit is proposed in Zhou et al. (2020) for single-branch
models (e.g., ResNet, Transformer). The work couples an internal classifier
with each layer of a pre-trained language model and dynamically stops
inference when the intermediate predictions of the internal classifiers remain
unchanged for a pre-defined number of steps. Experimental results with the
ALBERT model Lan et al. (2020) show that the technique can reduce the task
latency by up to 2.42$\times$ and slightly improve the model accuracy by
preventing it from overthinking and exploiting multiple classifiers for
prediction.

EEoI can statistically improve the latency of inference tasks by reducing the
inference workload at the price of a decrease in the accuracy. By increasing
throughput, the technique leads to better availability. The side branch
classifiers slightly increase the memory use during inference, while the task
computational efficiency is higher as in most of cases where side branch
classifiers can stop the inference earlier. In scenarios where a high level of
certainty is needed, an early exit might introduce a higher probability of
error, potentially compromising the reliability of the system. Generally, a
correctly designed and trained EEoI technique is able to improve energy
efficiency and optimize cost.

####  4.3.2 Inference Cache

Inference Cache saves models or models’ inference results to facilitate future
inferences of similar interest. This is motivated by the fact that ML tasks
requested by nearby users within the coverage of an edge node may exhibit
spatio-temporal locality Drolia et al. (2017). For example, users within the
same area might request recognition tasks for the same object of interest,
which introduces redundant computation of deep learning inference.

Besides the Cachier Drolia et al. (2017), which caches ML models with edge
server for recognition applications and shows 3$\times$ speedup in task
latency, DeepCache Xu et al. (2018) targets the cache challenge for a
continuous vision task. Given input video streams, DeepCache firstly discovers
the similarity between consecutive frames and identifies reusable image
regions. During inference, DeepCache maps the matched reusable regions on
feature maps and fills the reusable regions with cached feature map values
instead of real Convolutional Neural Network (CNN) execution. Experiments show
that DeepCache saves up to 47% inference execution time and reduces system
energy consumption by 20% on average. A hybrid approach, semantic memory
design (SMTM), is proposed in Li et al. (2021), combining inference cache with
EEoI. In this work, low-dimensional caches are compressed with an encoder from
high-dimensional feature maps of hot-spot classes. During the inference, SMTM
extracts the intermediate features per layer and matches them with the cached
features in fast memory: once matched, SMTM skips the rest of the layers and
directly outputs the results. Experiments with AlexNet, GoogLeNet Szegedy et
al. (2015), ResNet50, and MobileNet V2 Sandler et al. (2018) show that SMTM
can speed up the model inference over standard approaches with up to 2$\times$
and prior cache designs with up to 1.5$\times$ with only 1% to 3% accuracy
loss.

Inference cache methods show their advantages of reducing task latency on
continuous inference tasks or task batches. Since the prediction is usually
made together with current input and previous caches, the accuracy can drop
slightly. On the computational efficiency front, the cache lookup increases
the computing workload and memory usage, while the global computational
efficiency is improved across tasks, as the inference computation for each
data sample does not start from scratch. Energy consumption and cost are
reduced in the context of tasks sharing spatio-temporal similarity.

####  4.3.3 Model-Specific Inference Acceleration

Besides the above-mentioned edge inference techniques that can, in theory, be
applied to most of ML model structures, other research efforts aim at
accelerating the inference process for specific model structures. We briefly
review the representative methods of inference acceleration for three
mainstream neural network structures: (i) CNN, (ii) Recurrent Neural Network
(RNN), and (iii) Transformers.

For CNN models, MobileNets Howard et al. (2017) constructs small and low-
latency models based on depth-wise separable convolution. This factorizes a
standard convolution into a depth-wise convolution and a $1\times 1$
convolution, as a trade off between latency and accuracy during inference. The
latest version of MobileNets V3 Howard et al. (2019) adds squeeze and
excitation layers Hu et al. (2020) to the expansion-filtering-compression
block in MobileNets V2 Sandler et al. (2018). As a result, it gives unequal
weights to different channels from the input when creating the output feature
maps. Combined with a later neural architecture search and NetAdapt Yang et
al. (2018), MobileNets V3-Large reaches 75.2% accuracy and 156ms inference
latency on ImageNet classification with a single-threaded core on a Google
Pixel 1 phone. GhostNet Han et al. (2020) also uses a depth-wise convolution
to reduce the required high parameters and FLOPs induced by normal
convolution: given an input image, instead of applying the filters on all the
channels to generate one channel of the output, the input tensors are sliced
into individual channels and the convolution is then applied only on one
slice. During inference, $x$% of the input is processed by standard
convolution and the output of this is then passed to the second depth-wise
convolution to generate the final output. Experiments demonstrate that
GhostNet can achieve higher recognition performance, i.e., 75.7% better top-1
accuracy than MobileNets V3 with similar computational cost on the ImageNet
dataset. However, follow-up evaluations show that depth-wise convolution is
more suitable for ARM/CPU and not friendly for GPU; thus, it does not provide
a significant inference speedup in practice.

A real-time RNN acceleration framework is introduced in Dong et al. (2020) to
accelerate RNN inference for automatic speech recognition. The framework
consists of a block-based structured pruning and several specific compiler
optimization techniques including matrix reorder, load-redundant elimination,
and a compact data format for pruned model storage. Experiments achieve real-
time RNN inference with a Gated Recurrent Unit (GRU) model on an Adreno
640-embedded GPU and show no accuracy degradation when the compression rate is
not higher than 10$\times$.

Motivated by the way we pay visual attention to different regions of an image
or correlate words in one sentence, a transformer is proposed in Vaswani et
al. (2017) showing encouraging results in various machine learning domains
Dosovitskiy et al. (2020); Wang et al. (2022a). On the downside, transformer
models are usually slower than competitive CNN models Wang et al. (2022b) in
terms of task latency due to the massive number of parameters, quadratic-
increasing computation complexity with respect to token length, non-foldable
normalization layers, and lack of compiler-level optimizations. Current
research efforts, such as Graham et al. (2021); Roh et al. (2021), mainly
focus on simplifying the transformer architecture to fundamentally improve
inference latency, among which the recent EfficientFormer Li et al. (2022)
achieves 79.2% top-1 accuracy on ImageNet-1K with only 1.6ms inference latency
on an iPhone 12. In this work, a latency analysis is conducted to identify the
inference bottleneck on different layers of vision transformer, and the
EfficientFormer relies on a dimension-consistent structure design paradigm
that leverages hardware-friendly 4D MetaBlocks and powerful 3D multi-scale
hierarchical framework blocks along with a latency-driven slimming method to
deliver real-time inference at MobileNet speed.

Generally, model-specific inference acceleration techniques lower the workload
of an inference task and thus reduce the task latency within the same edge
environment, resulting in higher availability. Though computational resources
usage can vary among techniques, most work reports an acceptable accuracy loss
in exchange for a considerable decrease in resources usage. In the case of
model over-fitting, inference acceleration can improve the accelerated model
accuracy. The total energy consumption and cost are therefore reduced. Under
the assumption that the cached data are properly managed, the ML system
provides consistent responses to the same input, which can enhance the
reliability of the system.

##  5 Edge Learning

Edge learning techniques directly build ML models on native edge devices with
local data. Distributed learning, transfer learning, meta-learning, self-
supervised learning, and other learning paradigms fitting into Edge ML are
reviewed in this section to tackle different aspects of Edge ML requirements.

###  5.1 Distributed Learning

Compared to cloud-based learning in which raw or pre-processed data are
transmitted to cloud for model training, distributed learning (DL) in the edge
divides the model training workload onto the edge nodes, i.e., edge servers
and/or edge clients, to jointly train models with a cloud server by taking
advantage of individual edge computational resources. Modern distributed
learning approaches tend to only transmit locally updated model parameters or
locally calculated outputs to the aggregation servers, i.e., cloud or edge, or
the next edge node: in the server–client configuration, the aggregation server
constructs the global model with all shared local updates McMahan and Daniel
Ramage . On the other hand, in the peer-to-peer distributed learning setup,
the model construction is achieved in an incremental manner along with the
participating edge nodes together Wink and Nochta (2021).

Distributed learning can be applied to all three basic ML paradigms, namely,
supervised learning, unsupervised learning, and reinforcement learning.
Instead of learning from one optimization procedure $g_{\omega}$, distributed
learning constructs a global model by aggregating the optimization results of
all participant nodes, as formalized by Equation (8):

$\theta\approx\bigsqcup_{i=1}^{n}g_{\omega^{i}}(\mathbb{D}^{i},L_{\mathbb{D}^{i}})$
(8)

where $g_{\omega^{i}}$ is the optimization procedure driven by the meta-
knowledge ${\omega^{i}}$ of the participant node $i,i\\!\in\\!n$, and $n$ is
the number of distributed learning nodes. $\mathbb{D}$ stands for the data
used for learning, which can be for example the labelled dataset $D$ for
supervised learning, the unlabelled dataset $\bar{D}$ for unsupervised
learning, or the MDP $M$ for reinforcement learning. $L_{\mathbb{D}}^{i}$ is
the corresponding loss on the given data $\mathbb{D}$ and $\bigsqcup$ is the
aggregation algorithm (e.g., FedAvg Brendan McMahan et al. (2017) in the case
of Federated Learning) to update the model by the use of all participants’
optimization results (e.g., model parameters, gradients, outputs, etc.).

The edge distributed learning results into two major advantages:

  * •

Enhanced privacy and security: Edge data often contain sensitive information
related to personal or organizational matters that the data owners are
reluctant to share. By transmitting only updated model parameters instead of
the data, the distributed learning on the edge trains ML models in a privacy-
preserving manner. Moreover, the reduced frequency of data transmission
enhances the data security by restraining sensitive data only to the edge
environment.

  * •

Communication and bandwidth optimization: Uploading data to the cloud leads to
a large transmission overhead and is the bottleneck of current learning
paradigm Kang et al. (2017). A significant amount of communication is reduced
by processing data in the edge nodes, and bandwidth usage optimized via edge
distributed learning.

From the architectural perspective, there are three main organizational
architectures Abreha et al. (2022); Wang et al. (2020a) that exist to achieve
distributed learning in the server–client configuration, as illustrated in
Figure 3 and introduced as follows:

Figure 3: The distributed learning architectures available in the literature.

  * •

Cloud-enabled DL. Given a number of distributed and interconnected edge nodes,
cloud-enabled DL (see Figure 3a) constructs the global model by aggregating in
the cloud the local models’ parameters. These parameters are computed directly
in each edge device. Periodically, the cloud server shares the global model
parameters to all edge nodes so that the upcoming local model updates are made
on the latest global model.

  * •

Edge-enabled DL. In contrast to cloud-enabled DL, Edge-enabled DL (see Figure
3b) uses a local and edge server to aggregate model updates from its managed
edge devices. Edge devices, with the management range of an edge server,
contribute to the global model training on the edge aggregation server. Since
the edge aggregation server is located near the edge devices, edge-enabled DL
does not necessitate communications between the edge and the cloud, which thus
reduces the communication latency and brings task offline capability. On the
other hand, edge-enabled DL is often resource-constrained and can only support
a limited number of clients. This usually results in a degradation in task
performance over time.

  * •

Hierarchical DL. Hierarchical DL employs both cloud and edge aggregation
servers to build the global model. Generally, edge devices within the range of
a same-edge server transmit local data to the corresponding edge aggregation
server to individually train local models, and then local models’ parameters
are shared with the cloud aggregation server to construct the global model.
Periodically, the cloud server shares the global model parameters to all edge
nodes (i.e., servers and devices), so that the upcoming local model updates
are made on the latest global model. By this means, several challenges of
distributed learning, such as Non-Identically Distributed Data (Non-IID) Zhu
et al. (2021), imbalanced class Wang et al. (2021), and the heterogeneity of
edge devices Xu et al. (2021) with diverse computation capabilities and
network environments, can be targeted in the learning design. In fact, as each
edge aggregation server is only responsible for training the local model with
the collected data, the cloud aggregation server does not need to deal with
data diversity and device heterogeneity across the edge nodes.

In the following, we review two distributed learning paradigms in the context
of Edge ML: (i) federated learning, and (ii) split learning.

####  5.1.1 Federated Learning

Federated Learning (FL) McMahan and Daniel Ramage  enables edge nodes to
collaboratively learn a shared model while keeping all the training data on
edge nodes, decoupling the ability to conduct machine learning from the need
to store the data in the cloud. In each communication round, the aggregation
server distributes the global model’s parameters to edge training nodes, and
each node trains its local model instance with newly received parameters and
local data. The updated model parameters are then transmitted to the
aggregation server to update the global model. The aggregation is commonly
realized via federated average (FedAvg) Brendan McMahan et al. (2017) or
Quantized Stochastic Gradient Descent (QSGD) Alistarh et al. (2017) for neural
networks, involving multiple local Stochastic Gradient Descent (SGD) updates
and one aggregation by the server in each communication round.

FL is being widely studied in the literature. In particular, the survey in
Abreha et al. (2022) summarizes and compares more than forty existing surveys
on FL and edge computing regarding the covered topics. According to the
distribution of training data and features among edge nodes, federated
learning can be divided into three categories Yang et al. (2019): (i)
Horizontal Federated Learning (HFL), (ii) Vertical Federated Learning (VFL),
and (iii) Federated Transfer Learning (FTL). HFL refers to the federating
learning paradigm where training data across edge nodes share the feature
space but have different ones in samples. VFL federates models trained from
data sharing the sample IDs but different feature space across edge nodes.
Finally, FTL refers to the paradigm where data across edge nodes are
correlated but differ in both samples and feature space.

HFL is widely used to handle homogeneous feature spaces across distributed
data. In addition to the initial work of FL McMahan and Daniel Ramage ,
showing considerable latency and throughput when performing the query
suggestion task in mobile environments. HFL is highly popular in the
healthcare domain Xu et al. (2021) where it is, for instance, used to learn
from different electronic health records across medical organizations without
violating patients’ privacy and improve the effectiveness of data-hungry
analytical approaches. To tackle the limitation that HFL does not handle
heterogeneous feature spaces, the continual horizontal federated learning
(CHFL) approach Mori et al. (2022) splits models into two columns
corresponding to common features and unique features, respectively, and
jointly trains the first column by using common features through HFL and
locally trains the second column by using unique features. Evaluations
demonstrate that CHFL can handle uncommon features across edge nodes and
outperform the HFL models with are only based on common features.

As a more challenging subject than HFL, VFL is studied in Nock et al. (2018)
to answer the entity resolution question, which aims at finding the
correspondence between samples of the datasets and learning from the union of
all features. Since loss functions are normally not separable over features, a
token-based greedy entity-resolution algorithm is proposed in Nock et al.
(2018) to integrate the constraint of carrying out entity resolution within
classes on a logistic regression model. Furthermore, most studies of VFL only
support two participants and focus on binary class logistic regression
problems. A Multi-participant Multi-class Vertical Federated Learning (MMVFL)
framework is proposed in Feng and Yu (2020). MMVFL enables label sharing from
its owner to other VFL participants in a privacy-preserving manner. Experiment
results on two benchmark multi-view learning datasets, i.e., Handwritten and
Caltech7 Li et al. (2015), show that MMVFL can effectively share label
information among multiple VFL participants and match multi-class
classification performance of existing approaches.

As an extension of the federated learning paradigm, FTL deals with the
learning problem of correlated data from different sample spaces and feature
spaces. FedHealth Chen et al. (2020) is a framework for wearable healthcare
targeting the FTL as a union of FL and transfer learning. The framework
performs data aggregation through federated learning to preserve data privacy
and builds relatively personalized models by transfer learning to provide
adapted experiences in edge devices. To address the data scarcity in FL, an
FTL framework for cross-domain prediction is presented in Wang et al. (2022).
The idea of the framework is to share existing applications’ knowledge via a
central server as a base model, and new models can be constructed by
converting a base model to their target-domain models with limited
application-specific data using a transfer learning technique. Meanwhile, the
federated learning is implemented within a group to further enhance the
accuracy of the application-specific model. The simulation results on COCO and
PETS2009 Ferryman and Shahrokni (2009) datasets show that the proposed method
outperforms two state-of-the-art machine learning approaches by achieving
better training efficiency and prediction accuracy.

Besides the privacy-preserving nature of FL Li et al. (2021), and in addition
to the research efforts on HFL, VFL, and FTL, challenges have been raised in
federated learning oriented to security Lyu et al. (2020), communication Yang
et al. (2022), and limited computing resources Nguyen et al. (2021). This is
important as edge devices usually have higher task and communication latency
and are in vulnerable environments. In fact, low-cost IoT and Cyber-Physical
System (CPS) devices are generally vulnerable to attacks due to the lack of
fortified system security mechanisms. Recent advances in cyber-security for
federated learning Ghimire and Rawat (2022) reviewed several security attacks
targeting FL systems and the distributed security models to protect locally
residual data and shared model parameters. With respect to the parameter
aggregation algorithm, the commonly used FedAvg employs the aggregation server
to centralize model parameters; thus, attacking the central server breaks the
FL’s security and privacy. Decentralized FedAvg with momentum (DFedAvgM) Sun
et al. (2023) is presented on edge nodes that are connected by an undirected
graph. In DFedAvgM, all clients perform stochastic gradient descent with
momentum and communicate with their neighbors only. The convergence is proved
under trivial assumptions, and evaluations with ResNet-20 on CIFAR-10 dataset
demonstrate no significant accuracy loss when local epoch is set to 1.

From a communication perspective, although FL evades transmitting training
data over a network, the communication latency and bandwidth usage for weights
or gradients shared among edge nodes are inevitably introduced. The trade-off
between communication optimization and the aggregation convergence rate is
studied in Reisizadeh et al. (2020). A communication-efficient federated
learning method with Periodic Averaging and Quantization (FedPAQ) is
introduced. In FedPAQ, models are updated locally at edge devices and only
periodically averaged at the aggregation server. In each communication round
between edge training devices and the aggregation server, only a fraction of
devices participate in the parameters aggregation. Finally, a quantization
method is applied to quantize local model parameters before sharing with the
server. Experiments demonstrate a communication–computation trade-off to
improve communication bottleneck and FL scalability. Furthermore, knowledge
distillation is used in communication-efficient federated learning technique
FedKD Wu et al. (2022). In FedKD, a small mentee model and a large mentor
model learn and distill knowledge from each other. It should be noted that
only the mentee model is shared by different edge nodes and learns
collaboratively to reduce the communication cost. In such a configuration,
different training nodes have different local mentor models, which can better
adapt to the characteristics of local datasets to achieve personalized model
learning. Experiments with datasets on personalized news recommendations, text
detection, and medical named entity recognition show that FedKD maximally can
reduce 94.89% of communication cost and achieve competitive results with
centralized model learning.

Federated learning on resource-constrained devices limit both communication
and learning efficiency. The balance between convergence rate and allocated
resources in FL is studied in DInh et al. (2021), where an FL algorithm FEDL
is introduced to treat the resource allocation as an optimization problem. In
FEDL, each node solves its local training approximately till a local accuracy
level is achieved. The optimization is based on the Pareto efficiency model
Stiglitz (1982) to capture the trade-off between the wall-clock training time
and edge nodes energy consumption. Experimental results show that FEDL
outperforms the vanilla FedAvg algorithm in terms of convergence rate and test
accuracy. Moreover, computing resources can be not only limited but also
heterogeneous at edge devices. A heterogeneity-aware federated learning
method, Helios, is proposed in Xu et al. (2021) to tackle the computational
straggler issue. This implies that the edge devices with weak computational
capacities among heterogeneous devices may significantly delay the synchronous
parameter aggregation. Helios identifies each device’s training capability and
defines the corresponding neural network model training volumes. For
straggling devices, a soft-training method is proposed to dynamically compress
the original identical training model into the expected volume through a
rotating neuron training approach. Thus, the stragglers can be accelerated
while retaining the convergence for local training as well as federated
collaboration. Experiments show that Helios can provide up to $2.5\times$
training acceleration and maximum 4.64% convergence accuracy improvement in
various collaboration settings.

Table 3 summarizes the reviewed works related to FL topics and challenges.
Besides the efforts for security, communication, and resources, a personalized
federated learning paradigm is proposed in Hahn et al. (2022), so that each
client has their own personalized model as a result of federated learning. As
the existence of a connected subspace containing diverse low-loss solutions
between two or more independent deep networks has been discovered, the work
combines this property with the model mixture-based personalized federated
learning method for improved performance of personalization. Experiments on
several benchmark datasets demonstrated that the method achieves consistent
gains in both personalization performance and robustness to problematic
scenarios possible in realistic services.

Table 3: FL related work.
---
Data and Features for FL
HFL |  VFL |  FTL
McMahan and Daniel Ramage ; Brendan McMahan et al. (2017); Xu et al. (2021); Mori et al. (2022) |  Nock et al. (2018); Feng and Yu (2020) |  Chen et al. (2020); Wang et al. (2022)
Challenges
Enhanced Security |  Efficient Communication |  Optimized Resources
Lyu et al. (2020); Ghimire and Rawat (2022); Sun et al. (2023) |  Yang et al. (2022); Sun et al. (2023); Reisizadeh et al. (2020); Wu et al. (2022) |  Nguyen et al. (2021); DInh et al. (2021); Xu et al. (2021)

Overall, FL is designed primarily to protect data privacy during model
training. Sharing models and performing distributed training increases the
computation parallelism and reduces the communication cost, and thus reduces
both the end-to-end training task latency and the communication latency.
Moreover, specific FL design can provide enhanced security, optimized
bandwidth usage and efficient computing resource usage. The edge-enabled FL as
an instance of the edge-enabled DL can further bring offline capability to ML
models. Generally, since local devices can continue training on the local data
even if the network connection is down, which can improve the availability of
the application during network failures or disruptions. Implementation
requires the careful management and coordination of updates from multiple
devices, handling devices with differing computational capabilities, and
dealing with potential delays in communication. These factors can impact the
reliability of applications of Federated Learning.

####  5.1.2 Split Learning

As another distributed collaborative training paradigm of ML models for data
privacy, Split Learning (SpL) Gupta and Raskar (2018) divides neural networks
into multiple sections. Each section is trained on a different node, either a
server or a client. During the training phase, the forward process firstly
computes the input data within each section and transmits the outputs of the
last layer of each section to the next section. Once the forward process
reaches the last layer of the last section, a loss is computed on the given
input. The backward propagation shares the gradients reversely within each
section and from the first layer of the last section to the previous sections.
During the backward propagation, the model parameters are updated in the
meantime. The data used during the training process are stored across servers
or clients which take part in the collaborative training. However, none of the
involved edge nodes can review data from other sections. The neural network
split into sections and trained via SpL is called Split Neural Network (SNN).

The SpL method proposed in Gupta and Raskar (2018) splits the training between
high-performance servers and edge clients, and orchestrates the training over
sections into three steps: (i) training request, (ii) tensor transmission, and
(iii) weights update. Evaluations with VGG and Resnet-50 models on MNIST,
CIFAR-10, and ImageNet datasets show a significant reduction in the required
computation operations and communication bandwidth by edge clients. This is
because only the first few layers of SNN are computed on the client side, and
only the gradients of few layers are transmitted during backward propagation.
When a large number of clients are involved, the validation accuracy and
convergence rate of SpL are higher than FL, as general non-convex optimization
averaging models in a parameter space could produce an arbitrarily bad model
Goodfellow et al. (2015).

The configuration choice to split a neural network across servers and clients
are subject to design requirements and available computational resources. The
work in Vepakomma et al. (2018) presents several configurations of SNN
catering to different data modalities, of which Figure 4 illustrates three
representative configurations: (i) in vanilla SpL, each client trains a
partial deep network up to a specific layer known as the cut layer, and the
outputs at the cut layer are sent to a server which completes the rest of the
training. During the parameters update, the gradients are back-propagated at
the server from its last layer until the cut layer. The rest of the back
propagation is completed by the clients. (ii) In the configuration of SpL
without label sharing, the SNN is wrapped around at the end layers of the
servers. The outputs of the server layers are sent back to clients to obtain
the gradients. During backward propagation, the gradients are sent from the
clients to servers and then back again to clients to update the corresponding
sections of the SNN. (iii) SpL for vertically partitioned data allows multiple
clients holding different modalities of training data. In this configuration,
each client holding one data modality trains a partial model up to the cut
layer, and the cut layer from all the clients are then concatenated and sent
to the server to train the rest of the model. This process is continued back
and forth to complete the forward and backward propagation. Although the
configurations show some versatile applications for SNN, other configurations
remain to be explored.

Figure 4: Split Learning configurations Vepakomma et al. (2018).

Compared to FL, the SNN makes SpL a better option for resource-constrained
environments. On the other hand, SpL performs slower than FL due to the relay-
based training across multiple clients. To complement both learning paradigms,
Split Federated Learning (SFL) Thapa et al. (2022) aims at bringing FL and SpL
together for model privacy and robustness. SFL offers model privacy by network
splitting and client-side model updates based on SpL, as well as shorter
training latency by performing parallel processing across clients. Experiments
demonstrate that SFL provides similar test accuracy and communication
efficiency to SL, while significantly decreasing its computation time per
global epoch than in SpL for multiple clients.

Overall, SpL largely improves training task latency by taking advantage of
both server-side and edge-side computational resources. Compared to FL where
all model gradients or weights are transmitted over a network, SpL only shares
gradients of few layers of SNN and thus further optimizes the bandwidth usage.
By reducing the amount of the transmitted data, split learning can help
improve the availability of the application, especially in bandwidth-
constrained environments. The SNN model performance is better compared to FL
by avoiding FedAvg or QSGD during training. In addition to data privacy, which
is enhanced by all distributed learning paradigms, SpL is excellent at
preserving model privacy, as both the data and model structure are opaque
across sections. Energy consumption and cost are thus reduced as a result of
these SpL advantages. However, the implementation of Split Learning relies on
proper partitioning handling, device heterogeneity management, and
communication synchronisation, which can impact the reliability of the SpL
applications.

###  5.2 Transfer Learning

Transfer Learning (TL) is inspired by humans’ ability to transfer knowledge
across domains. Instead of training models from scratch, TL aims at creating
high-performance models on a target domain by transferring the knowledge from
models of a different but correlated source domain Panigrahi et al. (2021).
The knowledge transfer in the context of transfer learning can be in the
following three levels according to the discrepancy between domains:

  * •

Data Distribution. The training data obtained in a specific spatial or
temporal point can have different distribution to the testing data in an edge
environment. The different data distribution, due to different facts such as
co-variate shift Sugiyama et al. (2008), selection bias Huang et al. (2006),
and context feature bias Singh et al. (2020), could lead to the degradation of
model performance in a testing environment. The knowledge transfer between two
different data distributions is a subtopic of transfer learning, known as
Domain Adaptation (DA) Zhang et al. (2019).

  * •

Feature Space. Contrary to the homogeneous transfer learning Zhuang et al.
(2021) which assumes that the source domain and the target domain consist of
the same feature spaces, heterogeneous transfer learning tackles the (TL) case
where the source and target domains have different feature spaces Pan and Yang
(2010). The heterogeneous transfer learning applies a feature space adaptation
process to ease the difficulty to collect data within a target domain and
expands the transfer learning to broader applications.

  * •

Learning Task Space. Transfer learning also transfers knowledge between two
specific learning tasks by use of the inductive biases of the source task to
help perform the target task Weiss et al. (2016). In this level, the data of
the source and target task can have a same or different distribution and
feature space. However, the specific source and target tasks are supposed to
be similarly correlated either in a parallel manner, e.g., in the tasks of
objects identification and person identification, or in a downstream manner,
e.g., from a pretext learning task of image representation to a downstream
task of an object detection task. It is worth mentioning that the knowledge
generalization in an upstream manner from downstream tasks to out-of-
distribution data is Domain Generalization (DG) Zhou et al. (2021).

As a learning paradigm focusing on the techniques to transfer knowledge
between domains, transfer learning can be applied into all three basic
learning categories, i.e., supervised learning, unsupervised learning, and
reinforcement learning, for knowledge transfer between domains Weiss et al.
(2016). Based on the knowledge transfer process, two transfer learning
techniques exist to build neural networks for the target domain: (i) Layer
Freezing and (ii) Model Tuning. Layer Freezing is generally applied to
transfer knowledge between domains that are correlated in a parallel manner
and/or in situations where a target domain requests low training latency and
has few training data. The process is summarized as follows.

  1. 1.

Model Collection: An existing trained model on the source domain is acquired.

  2. 2.

Layer Freezing: The first several layers from a source model are frozen to
keep the previously learned representation, and the exact layers to freeze are
determined by the source model layers which have learned the source data
representation Li et al. (2018), i.e., usually the data-encoding part of a
model.

  3. 3.

Model Adjustment: The last few layers of the source model are deleted, and
again the exact layers to delete are determined by the source model structure
Zhi et al. (2017). New trainable layers are added after the last layer of the
modified source model to learn to turn the previous learned representation
into outputs on the target domain.

  4. 4.

Model Training: The updated model is trained with new data from the target
domain.

  5. 5.

Model Tuning: At last, an optional step is the tuning process usually based on
model fine-tuning Chu et al. (2016). During this step, the entire newly
trained model from the previous step is unfrozen and re-trained on the new
data from the target domain with a low learning rate. The tuning process
potentially further improves the model performance by adapting the newly
trained representation to the new data.

On the other hand, Model Tuning is generally applied to transfer knowledge
among domains that are correlated in a downstream manner and/or in situations
where a target domain has sufficient training data. The process of tuning
based transfer learning can be summarized as follows.

  1. 1.

Model Pre-training: A model is pre-trained on the source domain to learn
representations from source domain data.

  2. 2.

Model Adjustment: As an optional step in tuning process, the last few layers
of the source model are deleted, and new trainable layers are added after the
last layer of the modified source model.

  3. 3.

Model Tuning: The entire pre-trained model is trained on the new data from the
target domain to map the learned representation to the target output.

During the two transfer learning processes, the parameters of the original
model $\theta$ are updated to the new model parameters $\theta^{\prime}$ with
the dataset $\mathbb{D}^{\prime}$ from the target domain through an
optimization procedure $g_{\omega}^{\prime}$:

$\theta^{\prime}:=g_{\omega^{\prime}}(\mathbb{D}^{\prime},L_{\mathbb{D^{\prime}}})$
(9)

On the target domain, the meta-knowledge $\omega^{\prime}$ and the
optimization procedure $g_{\omega^{\prime}}$ can be derived from the source
domain during the transfer process; however, the focus of transfer learning is
the knowledge transfer of model parameters from $\theta$ to $\theta^{\prime}$.
Transfer learning building models based on previously learned knowledge in a
correlated domain brings the following benefits.

  * •

Training Efficiency. The speed of training new models is largely accelerated
and uses much less computational resources compared to model training from
scratch.

  * •

Less Training Data. The model training or tuning process on the target model
requires less training data, and this is especially useful in the case where
there is a lot of data available from the source domain and relatively less
data for target domain.

  * •

Model Personalization. Transfer learning can quickly specialize pre-trained
models to a specific environment and improve accuracy when the original pre-
trained model cannot generalize well.

Transfer learning techniques are studied and compared in several surveys: an
early study Pan and Yang (2010) associates the definition of transfer learning
to the reasoning-based categories, and divides transfer learning into: (i)
inductive transfer learning, (ii) transductive learning, and (iii)
unsupervised learning, with respect to the source and target task spaces. To
handle source and target feature space, homogeneous transfer learning is
reviewed in Zhuang et al. (2021); Weiss et al. (2016), and heterogeneous
transfer learning is analyzed in Pan and Yang (2010); Weiss et al. (2016).
Regarding the domain adaptation for different data distributions, the state-
of-the-art methods are summarized based on training loss in Wang and Deng
(2018) for computer vision applications. In particular, recent research
efforts tend to extend the scope of vanilla Domain Adaptation (DA) for
different data distributions to different feature spaces or task spaces. The
term “deep domain adaptation” is used in Wang and Deng (2018) to designate the
methods that leverage deep neural networks and DA to solve both distribution
shift and feature space differences. A Universal Domain Adaptation (UDA)
method is described in You et al. (2019) as a more general approach of
transfer learning across a task space. UDA targets the supervised model
transfer between domains where source and target have overlapped but have
different label spaces. Without prior knowledge on the label sets from both
domains, UDA is capable of classifying its samples correctly if it belongs to
any class in the source label set or mark it as “unknown” otherwise. To
address the unknown label classification, a Universal Adaptation Network (UAN)
is introduced to quantify the transferability of each sample into a sample-
level weighting mechanism based on both the domain similarity and the
prediction uncertainty of each sample. Empirical results show that UAN works
stably across different UDA settings and outperforms the state-of-the-art
closed set, partial and open set domain adaptation methods.

The related works of transfer learning are highlighted in Table 4 and are
introduced in the following. Regarding the layer freezing, one of the most
popular application domains is healthcare, as the training data related to
specific diseases can be difficult to obtain due to their rarity and the issue
of privacy. Transfer Learning is applied in Rios-Urrego et al. (2020) to
detect Parkinson’s disease from speech symptom with layer freezing. In this
work, the classification of patients with Parkinson’s disease is realized with
a CNN to analyze Mel-scale spectrograms in three different languages, i.e.,
Spanish, German, and Czech, via a transfer learning process. During the
knowledge transfer, several consecutive layers are frozen to identify the
layer topology characterizing the disease and others in the language. The
results indicate that the fine-tuning of the neural network does not provide
good performance in all languages, while the fine-tuning of individual layers
improves the accuracy by up to 7%. Moreover, transfer learning among languages
improves the accuracy up to 18% compared to a model training from scratch.
Since fine-tuning and storing all the parameters is prohibitively costly and
eventually becomes practically infeasible for pre-trained language models, a
parameter-efficient fine-tuning method is proposed in Ding et al. (2023) to
optimize a small portion of the model parameters while keeping the rest fixed,
drastically cutting down computation and storage costs.

Table 4: Highlighted related work of transfer learning.
---
Works |  Methods |  Insights
Rios-Urrego et al. (2020) |  Layer Freezing |  Frozen of consecutive layers to identify characterizing topology.
Ding et al. (2023) |  Parameter-Efficient Fine-tuning |  Optimization of a small portion of parameters.
Kara et al. (2021); Howard and Ruder (2018) |  Large Model Tuning |  Fine-tuning large pre-trained models.
Houlsby et al. (2019) |  Adapter Module Based Tuning |  Few trainable parameters added per task.
Lester et al. (2021) |  Prompt Tuning |  Competitive with scale, outperforms GPT-3’s few-shots learning.
Kojima et al. (2022) |  Prompt Tuning |  Increased zero-shot accuracy with “Let’s think step by step”.
Chung et al. (2022) |  Instruction Fine-tuning |  Large margin outperformance with FLan-PaLM 540B.
Schratz et al. (2019) |  Hyper-parameter Tuning |  Finding optimal model hyper-parameters.
Wang et al. (2019) |  Negative Transfer |  Adversarial network to filter unrelated source data.

Concerning the model-tuning, fine-tuning large pre-trained models is an
effective transfer mechanism in both CV Kara et al. (2021) and NLP Howard and
Ruder (2018) domains. As the general fine-tuning creates an entirely new model
for each downstream task, the method is not efficient when facing multiple
downstream tasks. In fact, it results in the reproduction of the same-sized
model multiple times. An adapter module-based tuning method is introduced in
Houlsby et al. (2019), where adapter modules extend the pre-trained models by
only adding a few trainable parameters per task. The parameters of the
original network remain fixed, yielding a high degree of parameter sharing.
The experiment transferring BERT transformer to 26 diverse text classification
tasks attain near state-of-the-art performance: on GLUE benchmark, the
proposed method shows only 0.4% degradation compared to fine-tuned results,
while adding only 3.6% parameters per task compared to the 100% parameter
retraining of fine-tuning. Moreover, prompt tuning Lester et al. (2021) is a
simple yet effective method to learn prompts to perform specific downstream
tasks without modifying models, which is especially useful when handling large
language models and vision–language models. The study in Lester et al. (2021)
shows that prompt tuning becomes more competitive with scale: as models exceed
billions of parameters, the proposed method matches the strong performance of
model fine-tuning, and largely outperforms the few-shots learning of
Generative Pre-trained Transformer 3 (GPT-3) Brown et al. (2020). As the
prompt plays an important role in the model output, an interesting discovery
is made in Kojima et al. (2022) to perform reasoning tasks with pre-trained
Large Language Models (LLMs) by simply adding “Let’s think step by step”
before each output. The zero-shot accuracy is increased from 17.7% to 78.7% on
MultiArith techmark Roy and Roth (2015) and from 10.4% to 40.7% on GSM8K
benchmark Cobbe et al. (2021) with an off-the-shelf 175B parameter model. As
explored by the work, the versatility of this single prompt across very
diverse reasoning tasks hints at untapped and understudied fundamental zero-
shot capabilities of LLMs. This suggests high-level and multi-task broad
cognitive capabilities may be extracted through simple prompting. Furthermore,
an instruction fine-tuning method is described in Chung et al. (2022) focusing
on scaling the number of tasks, scaling the model size, and fine-tuning on
chain-of-thought data. The resulting Flan-PaLM 540B instruction-fine tuned on
1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). At
last, the tuning process is also applied to find optimal values for model
hyper-parameters Schratz et al. (2019), which is however out of the scope of
transfer learning.

Although transfer learning depends on the correlation between source and
target domains to be effective, the similarities between domains are not
always beneficial but can be misleading to the learning. Negative transfer
Wang et al. (2019) is the transfer process in which the target model is
negatively affected by the transferred knowledge. It can be caused by several
factors such as the domain relevance and the learner’s capacity to find the
transferable and beneficial part of the knowledge across domains. The work in
Wang et al. (2019) proposes a method relying on an adversarial network to
circumvent negative transfer by filtering out unrelated source data. The
harmful source data are filtered by a discriminator estimating both marginal
and joint distributions to reduce the bias between source and target risks.
The experiments involving four benchmarks demonstrate the effectiveness of
filtering negative transfer and the improvement of model performance under
negative transfer conditions.

Transfer Learning avoids building models from scratch and largely reduces the
workload of training new models, which leads to the low training task latency,
availability improvement, and efficient computation. In parallel, the required
training data in the case of supervised learning is much less than required
when training models from scratch. Thus, transfer learning can save expensive
data-labeling efforts and drives conventional supervised learning more
independent of labelled data. Regarding the edge requirements of model
performance, transfer learning facilitates the construction of personalized
models specific to individual edge environments and are expected to maintain a
high model accuracy and reliability compared to generalized models. However,
in practice, the model performance is determined by the quality of the source
model, the training data in a target domain, and the correlation between the
source and the target domains. Thus, the performance varies according to the
specific configurations.

###  5.3 Meta-Learning

Taking the philosophy one step higher, and focusing on learning the learning
process rather than specific tasks, meta-learning Vanschoren (2018) is an
advanced learning paradigm that observes and “remembers” previous learning
experiences on multiple learning tasks, and then quickly learns new tasks from
previous meta-data by analyzing the relation between tasks and solutions. The
meta-learning solution for ML tasks is is realized in two levels Peng (2021):
(i) a base learner for each task, and (ii) a global meta-learner. The base
learner solves task-specific problems and focuses on a single task, while the
meta-learner integrates them using previous learned concepts to quickly learn
the associated tasks. For a new task, meta-learning directly applies or
updates the solution of the most similar task. In the case where no similar
task is registered, meta-learning exploits the relation between tasks and
solutions to propose an initial reference solution.

Meta-learning can also be applied to all three basic machine learning
paradigms: supervised learning, unsupervised, and reinforcement learning.
Regular supervised learning and unsupervised learning do not assume any given
or predefined meta-knowledge. On the contrary, in supervised and unsupervised
meta-learning, the goal is not only to realize a specific task but also to
find the best meta-knowledge set, enabling the base learner to learn new tasks
as quickly as possible. Regular reinforcement learning maximizes the expected
reward on a single MDP, while meta reinforcement learning’s intention is to
maximize the expected reward over various MDPs by learning meta-knowledge. To
summarize, instead of learning separately model parameters $\theta$ for all
base learners, meta-learning actually focuses on learning the optimal or sub-
optimal meta-knowledge $\omega^{\ast}$ for the global meta-learner, as
formalized in Equation (10).

$\omega^{\ast}:=\underset{\omega}{arg\,min}\
\bigsqcup_{t=1}^{n}L_{\mathbb{D}^{t}}(g_{\omega^{t}}(\mathbb{D}^{t},L_{\mathbb{D}^{t}}))$
(10)

where $g_{\omega^{t}}$ is the optimization procedure driven by the meta-
knowledge ${\omega^{t}}$ of the task $i,i\\!\in\\!n$, and $n$ is the number of
the considered base learner tasks. $\mathbb{D}^{t}$ is the data used for
learning the base task $t$, $L_{\mathbb{D}^{t}}$ is the corresponding loss on
the given data ${\mathbb{D}^{t}}$, $\bigsqcup$ is the aggregation algorithm
(e.g., Model- Agnostic Meta-Learning (MAML) Finn et al. (2017)) that finds the
optimal meta-knowledge $\omega^{\ast}$ by minimizing the losses across
different base learners.

Depending on the representation of the meta-knowledge, meta-learning
techniques can be divided into three categories Huisman et al. (2021): (i)
metric-based meta-learning, (ii) model-based meta-learning, and (iii)
optimization-based meta-learning.

  1. 1.

Metric-based meta-learning learns the meta-knowledge in the form of feature
space from previous tasks by associating the feature space with model
parameters. New tasks are achieved by comparing new inputs, usually with
unseen labels (also known as the query set), to example inputs (a.k.a. the
support set) in the learned feature space. The new input will be associated to
the label of the example input with which it shares the highest similarity.
The idea behind metric-based meta-learning is similar to distance-based
clustering algorithms, e.g., K-Nearest Neighbors (KNN) Zhang et al. (2017) or
K-means Hamerly and Elkan (2004), but with a learned model containing the
meta-knowledge. Being simple in computation and fast at test-time with small
tasks, metric-based meta-learning is inefficient when the tasks contain a
large number of labels to compare, while the fact of relying on labelled
examples makes the metric-based meta-learning both specialized at and limited
by the supervised learning paradigm.

  2. 2.

Model-based meta-learning relies on an internal or external memory component
(i.e., a model) to save previous inputs and to empower the models to maintain
a stateful representation of a task as the meta-knowledge. Specifically
designed for fast training, the memory component can update its parameters in
a few training steps with new data, either by the designed internal
architecture or controlled by another meta-learner model Santoro et al.
(2016). When given new data on a specific task, the model-based meta-learning
firstly processes the new data to train and alter the internal state of the
model. Since the internal state captures relevant task-specific information,
outputs can be generated for unseen labels of the same task or new tasks.
Compared to the metric-based meta-learning, model-based meta-learning has a
broader applicability to the three basic machine learning paradigms and brings
flexibility and dynamics to the meta-learning technique via quick and dynamic
model adjustment to new tasks and data.

  3. 3.

Optimization-based meta-learning revises the gradient-based learning
optimization algorithm so that the model is specialized at fast learning with
a few examples, as the gradient-based optimization is considered to be slow to
converge and inefficient with few learning data. Optimization-based meta-
learning is generally achieved by a two-level optimization process Finn et al.
(2017): base-learners are trained in a task-specific manner, while the meta-
learner performs cross-task optimization in such a way that all base learners
can quickly learn individual model parameters set for different tasks.
Optimization-based meta-learning works better on wider task distributions and
enables faster learning compared to the two previous meta-learning techniques.
On the other hand, the global optimization procedure leads to an expensive
computation workload, as each task’s base-learner is considered Hospedales et
al. (2022).

In all the three meta-learning representations, one important characteristic
of meta-learning is that during the testing phase, the resulting models are
generalized and able to deal with the data labels, inputs, and the tasks on
which models were not explicitly trained during the previous learning phase.
Thus, data and task generalization as well as fast learning are the two main
advantages of meta-learning.

Meta-learning widens the applicability of machine learning techniques and
hence is applied into various domains such as few-shot learning in image
classification Sun et al. (2021), zero-shot learning for natural language
processing Kojima et al. (2022), robot control Gupta et al. (2018), and
reasoning Griffiths et al. (2019). Several surveys study the existing meta-
learning techniques and works. In addition to Huisman et al. (2021), meta-
learning in neural networks is studied in Hospedales et al. (2022). This work
proposes a taxonomy and organizes the paper according to the representation of
meta-knowledge, the meta-level optimizer, and the global objective of the
meta-learning. Based on the type of the leveraging meta-data during the
learning process, Vanschoren et al. Vanschoren (2018) categorizes meta-
learning techniques into: (i) learning from model evaluations, (ii) learning
from task properties, and (iii) learning from prior models. Wang et al. Wang
et al. (2020) review the metric-based few-shot learning methods targeting the
problem of data-intensive applications with little training data. Methods are
grouped into three perspectives: data, model, and algorithm. The pros and cons
of each perspective is analyzed in the work. The specific related work on
Meta-Learning is summarized in Table 5 and described as follows.

The main challenge in meta-learning is to learn from prior experiences in a
systematic and data-driven way Vanschoren (2018). For the metric-based meta-
learning, a typical configuration of few-shot learning is N-way K-shot
learning Chen et al. (2019); Bennequin (2019). N-way refers to the number of
classes $N$ existing in the support set of the meta-testing phase. K-shot
refers to the number of data samples $K$ in each class in the support set. The
few-shot learning tackles the supervised learning problem where models need to
quickly generalize after training on few examples from each class. During the
meta-training phase, the training dataset is divided into support set and
query set, and the data embeddings are extracted from all training data, i.e.,
images. Each image from the query set is classified according to its embedding
similarity with images from the support set. The model parameters are then
updated by back-propagating the loss from the classification error of the
query set. After training, the meta-testing phase classifies unseen labels
from the meta-training phase (i.e., in Figure 5, images of unseen dog breeds
are given during meta-testing) by use of the new support set.

Table 5: Summary of related work on Meta-Learning.
---
Works |  Methods |  Results and Insights
Metric-based: Few-shot learning Chen et al. (2019); Bennequin (2019) |  N-way K-shot learning. |  Efficient for small tasks, but inefficient for large label sets.
Metric-based: LSTM-based meta-learner Ravi and Larochelle (2017) |  Captures both short-term and long-term knowledge. |  Rapidly converges a base learner to a locally optimal solution and in the meantime learns a task-common initialization as the base learner.
Metric-based: Zero-shot learning Romera-Paredes and Torr (2015) |  Two linear layers network modeling relationships among features, attributes, and classes. |  Outperforms state-of-the-art approaches on several datasets by up to 17.%
Metric-based: Class distribution learning Verma and Rai (2017) |  Represents each class as a probability distribution, defined as functions of the respective observed class attributes. |  Leverages additional unlabeled data from unseen classes and improves estimates of their class-conditional distributions; Superior results in the same datasets compared to Romera-Paredes and Torr (2015).
Metric-based: CLIP Radford et al. (2021) |  Pre-training of vision models from raw-text-describing images. |  Benchmarks on over 30 CV datasets produce competitive results with fully supervised baselines
Model-based: MANN Santoro et al. (2016) |  Model-based controller with an external memory component and Least Recently Used Access (LRUA) method. |  Superior to LSTM in two meta-learning tasks.
Model-based: Drone adaptation Belkhale et al. (2021) |  Dynamics model with shared dynamics parameters and adaptation parameters. |  Drone control with unknown payloads; autonomous determination of payload parameters and adjustment of flight control; performance improvement over non-adaptive methods on several suspended payload transportation tasks.
Optimization-based: MAML Finn et al. (2017) |  Gradient descent with model-specific updates. |  General optimization tasks; simple and general, but higher-order derivatives potentially decrease performance.
Optimization-based: iMAML Rajeswaran et al. (2019) |  Approximation of higher-order derivatives. |  General optimization tasks; more robust for larger optimization paths with same computational costs.
Optimization-based: online MAML Finn et al. (2019) |  Extension of MAML for online learning scenarios. |  Continuous learning from newly generated data; strong in model specialization, but computation cost grows over time.

The work Ravi and Larochelle (2017) proposes a Long Short-Term Memory
(LSTM)-based meta-learner model in the few-shot regime. This is done to learn
the exact optimization algorithm used to train another neural network
classifier as the base learner: the meta-learner is trained to capture both
short-term knowledge within a task and long-term common knowledge among all
the tasks. This way, the meta-learner is able to rapidly converge a base
learner to a locally optimal solution on each task and in the meantime learn a
task-common initialization as the base learner. As a step further, zero-shot
learning Wang et al. (2019) does not require any example data as support set
to perform new tasks or classify new classes which the model has not observed
during the training phase. A simple zero-shot learning approach is introduced
in Romera-Paredes and Torr (2015) to model the relationships among features,
attributes, and classes as a two linear layers network, where the weights of
the second layer are not learned but are given by the environment. During the
inference phase with new classes, the second layer is directly given so that
the model can make predictions on the new labels. Despite being simple, the
experiment results outperformed the state-of-the-art approaches on the
datasets of Animals with Attributes (AwA) Lampert et al. (2009), SUN
attributes (SUN) Patterson and Hays (2012), and aPascal/aYahoo objects (aPY)
Farhadi et al. (2009) by up to 17% at the publication time.

Unlike Romera-Paredes and Torr (2015) representing classes as fixed embeddings
in a feature space, Verma et al. Verma and Rai (2017) represent each class as
a probability distribution. The parameters of the distribution of each seen
and unseen class are defined as functions of the respective observed class
attributes. This allows the leveraging of additional unlabeled data from
unseen classes and the improvement of the estimates of their class-conditional
distributions via transductive or semi-supervised learning. Evaluations
demonstrate superior results in the same datasets compared to Romera-Paredes
and Torr (2015). In parallel to CV, the pre-trained large language models
(LLMs) have proven to be excellent for few-shot learner Brown et al. (2020)
and zero-shot learner Kojima et al. (2022). Furthermore, Contrastive Language-
Image Pre-training (CLIP) Radford et al. (2021) learns computer vision models
directly from raw-text-describing images, which leverages a much boarder
source of supervision instead of specific data labels. The pre-training of
predicting “which caption goes with which image?” is realized on a dataset of
400 million image and text pairs from the Internet. After pre-training,
natural language is used to reference learned visual concepts and describe new
ones, enabling zero-shot transfer of the model to downstream tasks. The work
matches the accuracy of the ResNet-50 model on ImageNet zero-shot without
dataset-specific training, and benchmarks on over 30 CV datasets produce
competitive results with fully supervised baselines.

Figure 5: N-way K-shot learning setup Bennequin (2019).

As for model-based meta-learning, Memory-Augmented Neural Network (MANN)
Santoro et al. (2016) contains a model-based controller, either feed-forward
network or LSTM, to interact with an external memory component for memory
retrieval and update. During training, the model learns to bind data
representations to their labels regardless of the actual content of the data
representation or label, and then the model maps these bound representations
to appropriate classes for prediction. The memory writing and reading are
powered by the proposed Least Recently Used Access (LRUA) method, and the MANN
displays a performance superior to an LSTM in two meta-learning tasks on the
Omniglot classification dataset Lake et al. (2019) and sampled functions from
a Gaussian process for regression.

A more concrete use case is illustrated in Belkhale et al. (2021) to adapt
drones to flight with unknown payloads, in which drones are expected to
autonomously determine the payload parameters and adjust the flight control
accordingly. During the training, a dynamics model with shared dynamics
parameters and adaptation parameters are trained over $K$ different payloads.
During the testing, the robot infers the optimal latent variable representing
the unknown payload by use of the learned dynamics parameters and the new
sensed data. A model-predictive controller (MPC) then uses the trained dynamic
model to plan and execute drone actions that follow the specified flight path.
Experiments demonstrate the performance improvement of the proposed method
compared to non-adaptive methods on several suspended payload transportation
tasks.

With respect to optimization-based meta-learning, MAML Finn et al. (2017) is a
general optimization algorithm, compatible with any model that learns through
gradient descent. In MAML, model-specific updates are made by one or more
gradient descent steps. Instead of using second derivatives for meta-
optimization of models, the meta-optimization proposes the First-Order MAML
(FOMAML) to ignore the second derivative during MAML gradient computation to
be less computationally expensive. MAML has obtained much attention due to its
simplicity and general applicability. In the meantime, ignoring higher-order
derivatives potentially decreases the model performance, and thus the iMAML
Rajeswaran et al. (2019) approximates these derivatives in a way that is less
memory-consuming. While the iMAML is more robust for larger optimization
paths, the computational costs roughly stay the same compared to MAML.
Furthermore, online MAML Finn et al. (2019) extends the MAML to online
learning scenarios where models continuously learn in a potentially infinite
time horizon from newly generated data and adapt to environmental changes.
Being strong in model specialization, the computation cost, however, keeps
growing over time.

Overall, meta-learning reduces supervised learning’s dependency on labelled
data by enabling models to learn new concepts quickly, which makes meta-
learning particularly suitable for the edge side in the sense that it
accelerates the training task. Another major advantage of meta-learning is the
generalization capability that it brings to models to solve diverse tasks and
the potential to realize general ML. Computational resource efficiency is
higher for multiple model training, which leads to optimized energy
consumption and cost optimization. Nevertheless, the global optimization
procedure of optimization-based meta-learning may yet lead to expensive
computation workload according to the number of base learners. Additional
computation on the support dataset for metric-based meta-learning introduces
extra workload during inference according to the dataset size (in such a case,
the use of metric-based meta-learning is usually avoided). The generalization
capability makes this application versatile and potentially more reliable, on
the one hand; while on the other hand, its resource-intensive nature may
impact its availability in resource-constrained environments.

###  5.4 Self-Supervised Learning

In contrast to supervised learning or reinforcement learning, human beings’
learning paradigm is barely supervised and rarely reinforced. Self-Supervised
Learning (SSL) is an unsupervised learning paradigm that uses self-supervision
from original data and extracts higher-level generalizable features through
unsupervised pre-training or optimization of contrastive loss objectives Peng
(2021). These learned feature representations are generalized and
transferable, and thus can be tuned later to realize downstream tasks, and the
pre-trained models are used as initial models to avoid training from scratch.
During self-supervised learning, data augmentation techniques Shorten and
Khoshgoftaar (2019); Shorten et al. (2021) are widely applied for contrast or
generation purposes, and data labels are not required since pseudo labels can
be estimated from trained models on similar tasks.

According to the loss objectives driving the training process, self-supervised
learning can be summarized into three categories Liu et al. (2021): (i)
generative learning, (ii) contrastive learning, and (iii) adversarial
learning, as a combination of generative and contrastive learning. The
architectures of the three categories are illustrated in Figure 6 to show the
transformation from the traditional supervised learning process to a self-
supervised learning approach, which can be particularly useful in Edge
scenarios where label data are scarce.

Figure 6: Self-Supervised Learning Architecture Liu et al. (2021).

  * •

Generative Learning: Generative learning trains an encoder to encode the input
into an explicit vector and a decoder to reconstruct the input from the
explicit vector. The training simulates pseudo labels for unlabeled data and
is guided by the reconstruction loss between the real input and the
reconstructed input.

  * •

Contrastive learning: Contrastive learning trains an encoder to respectively
encode inputs into explicit vectors and measure similarity among inputs. The
contrastive similarity metric is employed as the contrastive loss for model
training. During the training, the contrastive learning calibrates label-free
data against themselves to learn high-level generalizable representations.

  * •

Adversarial Learning: Adversarial learning trains an encoder–decoder to
generate fake samples and a discriminator to distinguish them from real
samples in an adversarial manner. In other words, it learns to reconstruct the