# Exploring the Intersection between Neural Architecture Search and Continual
Learning

Mohamed Shahawy1, Elhadj Benkhelifa2, and David White3  Smarts Systems, AI and
Cybersecurity Research Centre (SSAICS), Staffordshire University
Staffordshire, UK
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Despite the significant advances achieved in Artificial Neural Networks
(ANNs), their design process remains notoriously tedious, depending primarily
on intuition, experience and trial-and-error. This human-dependent process is
often time-consuming and prone to errors. Furthermore, the models are
generally bound to their training contexts, with no considerations to their
surrounding environments. Continual adaptiveness and automation of neural
networks is of paramount importance to several domains where model
accessibility is limited after deployment (e.g IoT devices, self-driving
vehicles, etc.). Additionally, even accessible models require frequent
maintenance post-deployment to overcome issues such as Concept/Data Drift,
which can be cumbersome and restrictive. By leveraging and combining
approaches from Neural Architecture Search (NAS) and Continual Learning (CL),
more robust and adaptive agents can be developed. This study conducts the
first extensive review on the intersection between NAS and CL, formalizing the
prospective Continually-Adaptive Neural Networks (CANNs) paradigm and
outlining research directions for lifelong autonomous ANNs.

###### Index Terms:

Continually Adaptive Neural Networks, Adaptive Neural Networks, Adaptive AI,
AutoML, Neural Architecture Search, Continual Learning, Incremental Learning

##  I Introduction

Deep Learning has led to notable advances across a number of applications,
specifically non-procedural tasks, such as image classiﬁcation or maneuvering
autonomous vehicles. By enabling computers to accomplish these tasks at a near
(or sometimes even surpassing) human-level performance [1], tasks that
previously relied on human-intervention can now become autonomous.

Despite all the incredible achievements accomplished in the field of
Artificial Intelligence, there remains a number of hurdles that keep AI models
from advancing further. Some Artificial Neural Networks (ANNs) have out-
performed humans at a number of tasks, such as playing chess [2], reading
comprehension [3], audio transcription [4], and several other areas [5, 6, 7,
8]. However, even the most advanced models still cannot compete with a child’s
cognitive abilities (reasoning, abstract thinking, complex comprehension,
etc.) [9, 10]. It is thought that human intelligence arises not only from
learned knowledge, but also through innate mechanisms that have evolved over
thousands of years [11].

Inspired by the Darwinian evolutionary theory, Neuroevolution is an area that
aims to evolves neural networks rather than develop them through classical
means. This evolutionary approach attempts to mimic the brain’s complex
structure and capabilities by generating populations of competing individuals
(neural architectures), essentially simulating the “survival of the fittest”
process. By doing so, the network topology and its weights can be encoded into
the optimization process and the best fitting combination of architectures and
parameters can be searched for in the solution space (as presented in the
iconic work by Stanley & Miikkulainen in 2002 [12]).

It was later discovered, however, that multi-objective optimization of the
neural architecture and its weights yielded less effective results than multi-
stage single-objective optimization (i.e. optimizing the architecture and its
parameters consecutively rather than concurrently) [13]. This AutoML area,
dubbed Neural Architecture Search (NAS), was formalized by Zoph & Le and
started trending in recent years [14]. The aim behind NAS is to leverage
optimization algorithms, such as Reinforcement Learning and Evolutionary
Algorithms, to provide more robust means of neural architecture design
relative to the classic, manual trial-and-error approach.

Another struggle modern AI faces lies in certain domains where the input
changes over the course of time. Typically, ANNs tend to partially or
completely overwrite learned knowledge upon retraining for new tasks, a
phenomenon referred to as Catastrophic Forgetting, or Catastrophic
Interference [15, 16]. To address this problem, the field of Continual
Learning (CL) was introduced (sometimes referred to as Lifelong Learning [17,
18, 19], (Task-/Domain-/Class-)Incremental Learning [20, 21, 22, 23], and
Sequential Learning [15, 24]).

Continual Learning aims to overcome Catastrophic Forgetting by retaining some
form of implicit or explicit memory within the model, or by having continuous
access to the tasks’ datasets. Although complete prevention of information
loss may not be attainable, the trade-off between generalizing the model to
adapt to a continuous input stream and consolidating learned knowledge can be
optimized. In other words, CL attempts to find a balanced solution on the
stability-plasticity Pareto front, maximizing shared learning representations
and adaptability to underlying non-stationary data distributions.

###  I-A Biological Inspiration

One of the reasons humans excel at continually adapting to new streams of
information without forgetting previously-attained knowledge is that our
experiences are generalized and largely overlap [25, 26]. In addition to the
human brain’s innate Continual Learning abilities, numerous other adaptive
qualities (synaptic plasticity, neuromodulation, neuroplasticity, etc.) have
been key components in the human evolution to intelligence [27, 28].

Research in the field of Neuropsychology has revealed that the evolutionary
advances accomplished by our species can be attributed to two fundamental
mechanisms: Associative Learning and Memory Formation [29].

Synaptic Plasticity, or the ability to change synaptic strengths, is thought
to underlie both the learning and memory mechanisms in mammalian brains [30,
31]. Hebb’s learning rule, which was postulated by Donald Hebb in 1949,
bridged the fields of psychology and neuroscience (now known as
neuropsychology) by modelling associative learning on a cellular level [32].
The premise behind the rule is that the learning process in biological neural
networks is activity-dependent; if a pre-synaptic neuron A is repeatedly
successful in activating a post-synaptic neuron B, the strength of the synapse
will gradually increase (typically referred to as Synaptic Potentiation).
Similarly, Synaptic Depression (or the weakening of synaptic strength) occurs
when a pre-synaptic neuron A consistently does not take part in firing a post-
synaptic neuron B. This learning model paved the way for one of the simplest
and earliest learning rules in Artificial Neural Networks.

The mechanism underlying memory formation and retention is considered one of
the most compelling and challenging research areas in the field of
neuroscience [33, 34]. Intuitively, it is thought that a particularly strong
connections between a set of neurons that resulted from high-frequency,
persistent strengthening of synapses is a direct representation of memory on a
cellular level. On the other hand, lack of access to those neurons would
result in a gradual decay of unused memories over time [35, 36]. It is worth
noting that researches on cognitive psychology have revealed that memory
entails a learning process in itself [37], and thus the mechanisms for both
processes are interleaved.

The Complementary Learning Systems framework, which was articulated by
McClelland et al. in 1995, defines the distinct roles of the mammalian
hippocampal and neocortical systems on learning and episodic memory [16]. The
theory holds that the hippocampus is responsible for rapid learning and
retaining short-term, sparse memories, which can later be replayed to the
Prefrontal Cortex for more efficient encoding and long-term storage. Although
the CLS is characterized as a theory, it has had over 25 years’ worth of
empirical data generally supporting its premise [38, 39].

The CLS model has inspired several Continual Learning approaches for ANNs,
often with explicit short-/long-term memories (modelling the Hippocampal
Complex/Medial Prefrontal Cortex) and a replay mechanism through a virtual
sleep phase [40, 41]. Although many other biological neural functions have not
been modelled in Artificial Neural Networks yet, there has been a growing
interest in fields pertaining to plasticity in computational models [17].

Figure 1: Comparative illustrations for Adaptive Learning and its neighboring
Machine Learning paradigms (inspired by [20])

###  I-B Motivation, Scope, and Contributions to Knowledge

Continually adaptive features in ANNs are becoming more imminent than ever;
several fields that have limited access to their deployed models are bound to
their fixed, pre-deployment states and cannot improve their functionality to
fit their changing surroundings. These fields include autonomous vehicles [42,
43, 44, 45, 46, 47, 48], robotics [49, 50, 51, 52, 53, 54], Edge and IoT
devices [55, 56, 57, 58], and numerous others [59, 60, 61, 62, 63, 64, 65].

In addition to the importance of Continual Learning in modern AI applications,
the classical process of developing neural networks is heavily-reliant on
manual human input; from defining a network architecture to tweaking
hyperparameters, several parameters are dependent on experts’ knowledge as
well as trial-and-error [66]. The AutoML field was introduced to facilitate
more robust Machine Learning solutions for non-experts by pipelining automated
optimization techniques, such as Neural Architecture Search (NAS) and
Hyperparameter Optimization. This area, however, is still in its infancy and
few studies have attempted to achieve an end-to-end self-developing ANN [67,
68].

Intuitively, the more statically-developed a neural network is (e.g. fixed
network topology, fixed training learning representation, etc.), the less
sensitive it will be to its surrounding environment. By minimizing or
completely eliminating manual input from the development of ANNs, a network
can become significantly less biased and thus more adaptive to fit the tasks’
contexts. This concept of self-managing autonomous agents has been explored in
the literature, however it is usually discussed within bounded domains (namely
Neuroevolution and Unsupervised Learning) [69, 17, 70].

Aims and Contributions. To expand the currently narrow and specialized mindset
associated with ANNs, we extend and generalize the inspiring ideas previously
proposed beyond the scope of unsupervised learning and Evolutionary
Algorithms. Continually-Adaptive Neural Networks (CANNs) aim to ultimately
eliminate human-intervention from the development and maintenance of ANNs by
combining Neural Architecture Search and Continual Learning solutions.
Functional to that aim, this study:

  1. 1.

Reviews Continual Learning approaches and proposes an improved categorization
scheme for memory models

  2. 2.

Reviews relevant works in the Neural Architecture Search field

  3. 3.

Explores the intersection between the two domains, formalizes the automated
lifelong supervised learning paradigm, and briefly reviews additional data
automation techniques (hyperparameter optimization and autonomous data
collection/pre-processing/feature engineering)

  4. 4.

Identifies the gaps and potential future directions for NAS, CL, and CANNs

Although there are several reviews conducted on Continual Learning [18, 20],
AutoML [71, 72, 73], and NAS [74, 75, 76, 66], the intersection between NAS
and Continual Learning has not been formalized or reviewed despite the
significant number of existing models in that area. To capture the scope of
this study, Fig. 1 visualizes the learning representation of Continual
Learning (task-incremental setting), Neural Architecture Search, their
intersection (CANNs), as well as neighboring paradigms to emphasize their
distinction.

Paper Outline. To investigate hybrid Continual Learning and Neural
Architecture Search models, we first conduct two distinct reviews on the
latest developments in each area (Sections II & III). After which, the
intersection of the two fields (dubbed the Continually-Adaptive Neural
Networks paradigm) is formalized, outlining ideal characteristics to achieve
autonomous and adaptive agents. Lastly, Section V summarizes the limitations
of the existing methodologies and delineates potential future research
directions.

##  II Continual Learning Approaches

Continual Learning aims to maximize the capacity for simultaneous solutions by
generalizing previous experiences to coincide with new tasks (a feature known
as Forward Transfer (FWT) of knowledge), whilst also consolidating and
maintaining previous knowledge (referred to as Backward Transfer (BWT)).
Balancing this trade-off between stability and plasticity is essential for a
sustainable CL model, and is commonly known as the Stability-Plasticity
dilemma [77, 78]. For instance, consider two incremental tasks,
$\mathcal{T}_{i}$ and $\mathcal{T}_{j}$, where
$\mathcal{T}_{i}\prec\mathcal{T}_{j}$.

Forward Transfer is the impact of $\mathcal{T}_{i}$ on the performance of
$\mathcal{T}_{j}$; if a preceding task is advantageous to the learning process
of a future task, it is considered positive FWT, which potentially leads to
“zero-shot”/“few-shot” learning [79]. Negative FWT, on the other hand, occurs
when a previous task hinders the learning process of a future task,
essentially making a model less pliable to fit new tasks.

Backward Transfer is the effect the learning process of task $\mathcal{T}_{j}$
has on the performance of task $\mathcal{T}_{i}$. There exists positive BWT if
learning $\mathcal{T}_{j}$ increases the performance of the preceding task
$\mathcal{T}_{i}$. Conversely, negative BWT exists when learning
$\mathcal{T}_{j}$ decreases the performance on $\mathcal{T}_{i}$ (i.e
Catastrophic Forgetting) [79, 18].

Since any model is constrained by a finite memory capacity and the number of
tasks is potentially unbounded, forgetting can be inevitable. As the number of
tasks increases, convergence becomes more compromised, leading to a decline in
the model performance. Therefore, the goal behind CL models is not to prevent
Catastrophic Forgetting, but to alleviate it and selectively minimize complete
loss of knowledge, a feature referred to as Graceful Forgetting [80, 81, 20,
82].

Types of Continual Learning. Given the incremental nature of Continual
Learning, there are three fundamental settings that may be encountered [23].
For a set of incremental tasks $\mathcal{T}$, each task $t$ can have $d\geq 1$
incremental datasets
$\mathcal{D}^{t}_{d}=\\{\mathcal{X}_{d},\mathcal{Y}_{d}\\}\>\;s.t.\>\;t\in\mathcal{T}$,
where $\mathcal{P}(\mathcal{X}),\mathcal{P}(\mathcal{Y})$ are the probability
distributions of the training data and output labels, respectively.

  1. (i)

Domain-Incremental Learning (DIL): A single task with dynamic input
distribution and a static label space.
$|\mathcal{T}|=1,\mathcal{P}(\mathcal{X}_{i})\neq\mathcal{P}(\mathcal{X}_{j}),\mathcal{P}(\mathcal{Y}_{i})=\mathcal{P}(\mathcal{Y}_{j}),\forall(i\neq
j)\in\mathcal{D}$

  2. (ii)

Class-Incremental Learning (CIL): A single task with dynamic input
distribution and an intersecting (but growing) label space. CIL is typically
modelled with an expanding single-head output layer.
$|\mathcal{T}|=1,\mathcal{P}(\mathcal{X}_{i})\neq\mathcal{P}(\mathcal{X}_{j}),\mathcal{Y}_{i}\cap\mathcal{Y}_{j},\forall(i\neq
j)\in\mathcal{D}$

  3. (iii)

Task-Incremental Learning (TIL): A set of distinct tasks with disjoint input
and output spaces. TIL is typically modelled with a multi-head output layer to
accommodate the dissimilar probability distributions of the tasks.
$|\mathcal{T}|>1,\mathcal{P}(\mathcal{X}_{i})\neq\mathcal{P}(\mathcal{X}_{j}),\mathcal{P}(\mathcal{Y}_{i})\neq\mathcal{P}(\mathcal{Y}_{j}),\forall(i\neq
j)\in\mathcal{D}$

In addition to the incremental settings, there are several different
approaches to retain memory in CL models. While there have already been a few
proposed taxonomies [83, 81, 20], we propose a categorization scheme (Fig. 2)
that adjusts the view of memory-retention models in the PANNs context.

Figure 2: Continual Learning memory-retention strategies

Previous works regarded “static/dynamic architectures” as a family of memory
models, whereas the adaptiveness of a network topology is tangent to
overcoming catastrophic forgetting. For example, some replay-based approaches
have fixed topologies [84], while others have dynamic topologies [85]. This
showcases that structural plasticity is a catalyst for memory improvement that
can be combined with any strategy in Fig. 2, rather than a holistic approach
for retaining knowledge. Hence, we propose to classify structural plasticity
as an independent quality that, albeit relevant to memory-retention, should
not be categorized as a distinct family of CL models.

###  II-A Replay-Based Methods

The first of the two families of methods is Replay-Based Continual Learning.
Replay-Based models retain an episodic memory by storing representations that
best describe the distribution of the data used for each task. These
representations could simply be a raw portion of the datasets (Episodic
Rehearsal), or it could be the model itself that was trained on a previous
task used to infer and generate the samples used (essentially reverse-
engineering the model) (Generative Rehearsal, often called pseudo-rehearsal).
Each sub-type of the Replay-based models has its own strengths and weaknesses:

  * •

Episodic Rehearsal: store a subset of the datasets used to later ”rehearse”
them in subsequent training of newer tasks. While this method might be the
easiest to implement out of the 2 sub-types, the model can easily overfit to
the subsets due to the limited number of samples, and does not scale well in
terms of memory cost.

  * •

Generative Rehearsal: store the previous tasks’ distributions instead of the
raw samples, which can later be used to project and generate auxiliary data
for replay purposes. Although this method has not been used frequently in
recent studies on CL, the recent emergence of generative networks opens a
potential for new Generative Rehearsal models [20].

Both Replay-Based approaches tend to be computationally- and/or memory-
expensive, especially as the number of tasks increases [84, 18, 86, 20].
However, by caching portions of the datasets or the models used for each task,
the retraining process can support prioritization for particular tasks,
thereby facilitating more control over Graceful Forgetting.

In general, Replay-Based methods draw inspiration from the biological CLS
framework, where short-term memory is quickly stored to the hippocampus, to be
later learned slowly and in a more generalized fashion by the neocortex
(likely during sleep). For a more comprehensive review on the association
between biological mechanisms and Replay-Based CL models, the recent work by
Hayes et al. covers the topic extensively and in-depth [87].

###  II-B Parameter-Constraint Methods

The latter family of approaches is Parameter-Constraint Continual Learning. By
constraining parameters through isolation, regularization, or any form of
transfer learning/fine-tuning, Parameter-Constraint models can efficiently
prevent Catastrophic Forgetting whilst not wasting memory on prior data
storage. Sub-types of Parameter-Constraint models are split three-fold:

  * •

Data-Focused: constrain parameters through a combination of Fine-Tuning and
Knowledge Distillation [88], where a temporary (large and highly regularized)
model is trained on a single task and is then used to transfer the knowledge
to the main (smaller) network.

  * •

Prior-Focused: constrain parameters through regularization. By penalizing
changes to the most influential parameters in the network, Catastrophic
Forgetting can be minimized. Geometrically, the model attempts to find the
area of best fit (where most solutions coincide) in the solution space.

  * •

Parameter-Isolation: constrain parameters through isolation. Unlike Prior-
Focused models, this method completely isolates some of the weights
($\lambda=0$) instead of just applying penalties, hence dedicating portions of
the network to individual tasks.

A minor limitation with penalizing changes to the weights or completely
isolating them is that the priority will typically be given to the earlier
tasks, rendering Graceful Forgetting less attainable using traditional methods
[89, 90].

By combining Replay-Based and Parameter-Constraint methods, a third, implicit
family of approaches can be defined. The first method to include
regularization techniques on top of an episodic memory is Gradient Episodic
Memory for Continual Learning (GEM) [79]. GEM aims to constrain the parameter
changes pertaining to new tasks by placing an upper-bound limit on the losses
of each task; the loss of every task can decrease but cannot increase.

Model |  Incremental Setting |  Memory Strategy |  Limitations
---|---|---|---
Learning Without Forgetting (LwF) [91] |  Task-Incremental |  Data-Focused |  • Highly dependent on the relevance of the tasks • Training time gradually increases with the number of tasks • Architecture grows linearly with each task-increment
Incremental Classifier and Representation Learning (iCaRL) [84] |  Class-Incremental |  Episodic Rehearsal |  • Requires explicit storage for older tasks • Can easily overfit on the tasks’ exemplars subset • The exemplars chosen may not be representative of the entire dataset
Elastic Weight Consolidation (EWC) [92] |  Task-Incremental |  Prior-Focused |  • Strictly requires a fixed network architecture (known in advance) • Numerically unstable for a large number of tasks
Gradient Episodic Memory for Continual Learning (GEM) [79] |  Task-Incremental |  Episodic Rehearsal & Prior-Focused |  • Requires explicit storage for older tasks • Highly dependent on the relevance of the tasks • No memory-management policy
Memory-Aware Synapses (MAS) [80] |  Task-Incremental |  Prior-Focused |  • Strictly requires a fixed network architecture (known in advance)
Progressive Neural Networks (PNN) [89] |  Task-Incremental |  Parameter-Isolation |  • Does not facilitate positive BWT
Experience Replay for Continual Learning (CLEAR) [86] |  Task-Incremental |  Episodic Rehearsal |  • Built on the assumption that input data stream is always iid
Continual Learning through Synaptic Intelligence (SI) [93] |  Task-Incremental |  Prior-Focused |  • Inefficient when applied to a pretrained network • Tasks’ order affects performance • The importance of the weights is not properly estimated in some cases (as noted by the authors)
Deep Generative Dual Memory Network for Continual Learning (DGR) [94] |  Task-Incremental |  Generative Rehearsal |  • Strictly requires a fixed network architecture (known in advance) • Model saturates with tasks over time (performance declines as the number of tasks increases)
Incremental Moment Matching (IMM) [95] |  Class-Incremental & Domain-Incremental |  Prior-Focused |  • Requires explicit storage for previous tasks’ models
Less-Forgetting Learning (LFL) [96] |  Domain-Incremental |  Data-Focused |  • Highly dependent on the relevance of the tasks • Susceptible to data-distribution (domain) shift [97]
Expert Gate [90] |  Task-Incremental |  Parameter-Isolation |  • Does not facilitate positive BWT due to gating mechanism

TABLE I: Brief analysis of Continual Learning models from an adaptiveness
perspective

###  II-C Evaluation of Continual Learning Performance

Despite the large body of research on Continual Learning, the benchmark tests
and evaluation metrics for Continual Learning models are largely heterogeneous
and very few works have been devoted to formalize an evaluation framework that
captures the models’ performance in dynamic environments [89, 18, 50, 98]. The
majority of the early works relied on a rather uninformative evaluation
approach, measuring only the average accuracy across all tasks. While accuracy
is generally considered a reliable measure of the overall model performance,
Continual Learning should be ranked based on more robust criteria that are
specific to dynamic scenarios.

Lopez-Paz & Ranzato proposed one of the first formal evaluation methods for
Continual Learning models [79]. Their methodology revolves around more ”human-
like” test environments, where the models’ evaluation should be based on a
large number of tasks, each with a relatively small number of examples, and
the reported metrics should include not only accuracy, but also a measure of
forward/backward transfer. Given that some models can yield the same
accuracies, FWT and BWT can provide a better insight on the model’s long-term
performance. Following the same direction, Díaz-Rodríguez et al. later
proposed a similar evaluation framework, with the addition of 3 metrics:
model-size efficiency, samples storage-size efficiency, and computational
efficiency [99]. While these metrics are not directly related to the tasks’
performance (i.e accuracy, stability, plasticity, etc.), they are vital
indicators to a model’s scalability.

While the measure of FWT and BWT can provide extensive information on the
behavior of a model, they do not directly estimate how well a model retains
memory. Kemker et al. focused on assessing a model’s ability to overcome
Catastrophic Forgetting particularly [100]. The study establishes 3
(normalized AUC-ROC curve) metrics that jointly measure memory retention, the
model’s ability to acquire new knowledge, and the model’s overall performance:
$\Omega_{base}$, $\Omega_{new}$, and $\Omega_{all}$. The base metric evaluates
how well the first/base task is retained after subsequent tasks have been
learned. $\Omega_{new}$ measures the performance of a new task immediately
after it is learned, which shows the model’s few-shot learning ability.
Lastly, $\Omega_{all}$ evaluates the model’s overall performance in terms of
retention and acquisition of knowledge.

Inspired by the aforementioned approach, Hayes et al. proposed a similar
evaluation framework that is split into 3 experimental paradigms [101]. The
first benchmark test simply measures the model’s ability to quickly learn new
tasks through continual iid data. While real-world scenarios tend to comprise
of non-iid streams of data, this test is used to establish a baseline metric
that can be compared with offline models. The second test evaluates the
model’s ability to incrementally learn new classes whilst measuring the tasks’
test accuracies with each increment, which can also indicate when Catastrophic
Forgetting occurs. Finally, the last test consists of a stream of organized
non-iid data, resembling a realistic stream of stimuli that could be
experienced by a deployed model. This framework also uses the $\Omega_{all}$
metric to measure stability, plasticity, and overall performance.

It is important to highlight that Continual Learning environments can vary to
great extents. A robust evaluation framework should accommodate these
variations and yield consistent metrics regardless of the application’s
environment. Farquhar & Gal introduced the first desiderata for CL ranking,
highlighting the desirable features in an evaluation framework [83]. The
proposed desiderata suggests that data from later tasks should resemble
earlier tasks to some extent, arguing that some evaluation approaches use
permuted datasets (permuted MNIST in particular [100, 101]), which is an
unnecessarily excessive test. The study also claims that the output vector
used in the training phase should be the same as the one used during testing.
Another desirable feature in the evaluation of a CL model is to restrict
retraining for older tasks; the model should not be allowed unconstrained
access to early tasks’ examples. Last but not least, the test should include
more than 2 tasks, as success on such a small sample size cannot yield
reliable performance metrics that can be applicable to larger scales.

The most recent formal study on Continual Learning evaluation metrics was
proposed by New et al. in 2022 as part of the DARPA L2M research program [98].
The primary aim behind this evaluation framework is to robustly rank models in
a domain-agnostic fashion, doing so based on 5 metrics: Performance
Maintenance (across all tasks), FWT, BWT, Performance Relative to a Single
Task Expert, and Sample Efficiency (ability to improve the learning process
over time). Together, these metrics capture the effectiveness of memory-
retention, knowledge transfer & adaptation, and model sustainability. Not only
do such metrics assist researchers in identifying the suitable models for each
application domain, they also empirically consolidate all the ideal qualities
for CL models proposed in previous studies. Aiming for higher ranks on this
domain-agnostic adaptiveness scale further paves the way towards autonomous
agents.

##  III Neural Architecture Search Approaches

Most popular Deep Neural Networks are designed by human experts, where the
potentially hundreds of layers are manually specified [102, 103, 104, 105].
The architecture design process usually depends on the experts’ educated
intuition and a tedious trial-and-error process. As the tasks become more
complex, the networks are normally expected to be deeper and thus their
architectures become trickier to design and optimize.

In addition to the extensive knowledge required to design neural
architectures, humans tend to have a fixed thinking paradigm that limits the
exploration of the search space and leads to unvaried solutions. These
drawbacks of manual Deep Neural Networks’ design were further recognized when
an automatically-tuned neural network (Auto-Net 1.0) won against human
experts’ designs in the ChaLearn AutoML challenge in 2015 [106, 107, 108].

The field of Neural Architecture Search (NAS) emerged to minimize or
completely eliminate the limitations posed by the human factor in neural
architecture-design whilst providing more robust means of finding optimal
topologies. The NAS process comprises of defining 3 main components (as
depicted in Fig. 3): a Search Space, a Search Algorithm, and an Evaluation
Strategy [74, 66].

Figure 3: The general Neural Architecture Search framework

###  III-A Search Spaces

The first step in a typical NAS approach is defining the search space. The
search space dictates the potential solutions’ boundary and can have a
detrimental impact on the outcome of the entire search process [109]. An over-
simplified search space will likely lead to shallow networks that are only
capable of solving simple tasks, while a non-continuous and high dimensional
space can yield more complex networks, but is difficult to optimize and is
typically computationally expensive.

####  III-A1 Layer-Wise Space

One of the simplest search spaces is the Layer-Wise space (sometimes referred
to as the Stage-Wise or Chain-Structured search space [110, 111, 66, 112]), as
seen in the iconic early works of Zoph & Le and Baker et al. [14, 113, 74]. As
implied by the name, the Layer-Wise space is used to sample a sequentially-
built network from a space of layers. For a simple Layer-Wise architecture
$\mathcal{A}$ consisting of $n$ layers, the $i^{th}$ layer $L_{i}$ would
receive its input from $L_{i-1}$, and its output would feed into $L_{i+1}$,
forming the chain $\mathcal{A}=L_{n-1}\circ\dots\circ L_{1}\circ L_{0}$ (as
illustrated in Fig. 4).

Figure 4: Layer-Wise Search Space for Neural Architecture Search; A single-
branch architecture sample (top) and a multi-branch counterpart (bottom)

A Layer-Wise search space can be defined through 3 parameters: The number of
layers $n$, the type of layers (e.g. convolution, pooling, flattening, the
number of units (in a fully-connected network), etc.), and the hyperparameters
associated with the previously-defined layer types. Some studies, however,
have more recently started introducing more complex design elements, such as
Skip-Connections (depicted in Fig. 4; bottom) [113, 14, 114, 115, 74, 116].
The addition of Skip-Connections allows the search algorithms to build multi-
branch networks (similar to ResNet [104] and DenseNet [117]) which
significantly expands the search space. This expansive nature, however, is
inherently greedy and resource-intensive.

####  III-A2 Cell-Based Space

To circumvent the high performance requirements posed by the Layer-Wise space,
more recent works took advantage of the repeated fixed sub-structures in
neural networks, creating cells (i.e groups of operations; see Fig. 5) that
can be reused as individual units. This modular approach to neural
architecture design was manually implemented in some popular networks, such as
ResNet and InceptionNet [102, 104], and is also one of the most common search
spaces in NAS models. The Cell-based Space (often referred to as the Modular
Search Space and the Block-Based Space) drastically reduces the resource cost
by compounding the building-blocks that can be used [118, 115, 119, 74, 66].

Figure 5: Generic Cell-based Neural Architecture Search topology sample

Zoph et al. introduced one of the top-performing popular NAS frameworks,
Google’s NASNet, which was developed using a Cell-Based NAS space [115]. The
NASNet Search Space comprises of only 2 types of cells: a Normal Cell, and a
Reduction Cell. The Normal Cell performs feature extraction whilst preserving
the input dimensionality, whereas the Reduction Cell down-samples the given
feature map to half its size. By manually defining a fixed macro-architecture
(the structure comprising of Normal/Reduction cells), optimal operations for
each cell-type can be searched for. By adopting this higher level design
paradigm and shifting the manual design focus from an elementary-unit level to
the macro-architecture, several NAS frameworks have attained exceptional
performance results [115, 119, 114, 120, 121, 122].

In addition to the reasonable resource usage the Cell-based Space offers, the
modular nature of the space enables the transferability of cells back and
forth between networks (i.e cell-based Transfer Learning). For instance,
individual cells from NASNet were optimized on CIFAR-10, then transferred and
used on ImageNet, breaking highest-accuracy records on both benchmarks (97.60%
and 82.70%, respectively) [115].

Relative to the Layer-Wise space, this approach breaks down the optimization
phase into smaller problems that can be computed faster. However, this reduced
search-cost is leveraged at the expense of the agility of the framework. The
Cell-based space requires a priori knowledge to manually pre-define the static
macro-structure that constrains the search space, as opposed to the more
expansive Layer-Wise space that can dynamically adapt based on the tasks’
context.

####  III-A3 Hierarchical Space

Another search space that also focuses on macro-architecture optimization is
the Hierarchical Space, first introduced by Liu et al. [123]. Similar to the
aforementioned modular approach, the Hierarchical Space’s concept is to
construct sub-graphs of a topology, then use them to recursively compose an
architecture. The key difference between the Cell-Based and Hierarchical
spaces is that the latter uses recursive composition to build the topology
rather than modular stacking.

At the core, the Hierarchical Space uses primitive operations (e.g. 1x1
convolution, 3x3 maxpooling, ReLU, etc.) to set up the first level,
$\ell_{1}$. The next level $\ell_{2}$ would consist of sub-graphs, each
composed of multiple primitive elements from $\ell_{1}$, and so on for each
subsequent level.

Furthermore, the composition process yields a new operation in the form of a
Directed Acyclic Graph (DAG), which are typically encoded as upper-triangular
adjacency matrices. More formally, each level $\ell$ in the Hierarchical Space
comprises of $n$ adjacency matrices $\mathcal{O}^{(\ell)}=\langle
o^{(\ell)}_{1},o^{(\ell)}_{2},...,o^{(\ell)}_{n}\rangle$, such that:

$o^{(\ell)}_{i}=compose(\mathcal{O}^{(\ell-1)})\>,\;\forall\ell=2,...,L$ (1)

Where the $compose(\cdot)$ function constructs an operation using a
combination of multiple motifs from the given set of operations (the previous
level). The Hierarchical Space has proven to be an efficient search space,
even when optimized by naïve Search Algorithms, such as Random Search and Grid
Search [123, 74].

####  III-A4 Other Search Spaces

Earlier works on neural architecture optimization, particularly using
Neuroevolution and other Metaheuristic algorithms, searched for optimal
Perceptrons on a synaptic/neural level (to be referred to as the Neuronal
Search Space) [12, 124, 125]. This granular approach, however, is generally
limited to shallower and simpler topologies relative to modern NAS spaces
[13].

Although the Layer-Wise, Cell-Based, and Hierarchical search spaces are
considered some of the most commonly used spaces in the literature, there has
been a surge of alternative methodologies recently. For instance, rather than
manually designing a static search space, Ci et al. proposed a novel approach
to evolve progressively-growing search spaces, which they call Neural Search-
space Evolution (NSE) [126]. The authors of NSE claim that traditional search
space-design approaches cannot effectively handle large spaces, stating that
using existing state-of-the-art algorithms could take up to 27,000 (Nvidia GTX
1080Ti) GPU hours for 100 epochs’ worth of search through a large,
27-operations space. However, using NSENet, that number diminishes to 4,000
GPU hours, with slightly better Top-1 accuracy score.

Due to the NAS Search Spaces’ discrete nature, Gradient Optimization was
inherently not considered a valid optimization approach. However, Liu et al.
introduced the concept of relaxing the structural parameters of the network,
transforming the discrete search space into a continuous and differentiable
space (often denoted as the Continuous Search Space) [127]. This relaxation
process consists of transforming all categorical operations into a continuous
choice of probabilities. A particular operation $\bar{o}_{(i,j)}$ is relaxed
using softmax over the set of all operations in the space $\mathcal{O}$ as
follows:

$\bar{o}_{(i,j)}(x)=\sum_{o\in\mathcal{O}}\frac{e^{\alpha_{o}(i,j)}}{\sum_{o^{\prime}\in\mathcal{O}}e^{\alpha_{o}(i,j)}}o(x^{(i)})$
(2)

Another unique continuous Search Space-design approach is Memory-Bank
Representation (MBR), introduced by Brock et al. [128]. This method uses an
auxiliary fully-convolutional “HyperNet” to produce weights for randomly
sampled architectures (the memory banks). In addition to dramatically reducing
the candidates’ training time through the HyperNet’s One Shot evaluation
(expanded upon in subsection III-C5), this proxy representation enables
approximate differentiation, which supports architectural gradient descent.

###  III-B Search Algorithms

After defining a Search Space, the next step in a standard NAS pipeline is to
create and optimize a structure using a Search Algorithm. A Search Algorithm
is the approach used to sample the candidate network(s) from a given solution
space to be evaluated thereafter and further optimized if needed. Most NAS
Search Algorithms are similar to approaches used for hyperparameter
optimization as both areas are inherently similar [129].

####  III-B1 Random/Grid Search

Although widely considered a naïve optimization approach due to its exhaustive
nature, Random Search (RS) has proven to be useful for hyperparameter tuning
[130, 131]. RS algorithms for NAS work by selecting a candidate network at
random (using a given distribution, e.g.: uniform, normal, poisson, etc.) from
the search space [132]. The glaring downside of such an approach is that the
selection process is statistically independent and cannot be optimized given
the feedback from the Evaluation Strategy. However, the lack of complexity
leads to a much faster selection process and therefore great exploration
capabilities, which can yield high-performing results [123, 109].

Another popular exhaustive search technique used for Neural Architecture
Search is Grid Search (GS). By defining the number of layers and a set of
operations per layer
(i.e $L_{i}=\\{conv3x3,maxpool3x3,dense256\\}$), GS can select network
architectures by trying different combinations of options from the given sets
of operations. As with Random Search, the values are independent of each other
in Grid Search approaches, meaning the algorithm can be parallelized. However,
out of the two exhaustive algorithms, Random Search often outperforms Grid
Search [129].

####  III-B2 Reinforcement Learning

Reinforcement Learning (RL) problems generally comprise of (i) an Agent that
uses an optimization algorithm to take certain actions and achieve a given
task, (ii) a Policy that defines the set of possible actions, (iii) an
Environment (sometimes called an Action/Observation Space) that the Agent can
interact with, and finally (iv) a Reward Function that incentivizes the Agent
every iteration to reach an optimal solution. When applied to NAS, these
components remain the same; the architecture generator/controller (the Agent;
typically an RNN controller model) uses a set of possible operations/layers
(the Policy) in a given Search Space (the Environment), which can then be
assessed using an Evaluation Strategy (the Reward Function).

NAS is widely considered to have gained popularity after the groundbreaking
work by Zoph & Le in 2017, which was one of the earliest models to achieve
impressive performance on the CIFAR-10 and the Penn-Treebank datasets using an
automatically-searched neural architecture [14]. Their model was based on a
Reinforcement Learning search algorithm that used a Recurrent Neural Network
(RNN) with the REINFORCE Policy Gradient algorithm [133] as an
Agent/controller that samples a Convolutional Neural Network (CNN) from a
given set of CNN operations (convolutional layers with a variable number of
kernels and kernel sizes, ReLU, Batch Normalization, and Skip-Connections).
Some of the generated models using this approach achieved an error-rate of
3.65% on the CIFAR-10 image dataset, outperforming ResNet and some variations
of DenseNet. The highly criticized aspect of this approach is its vast
computational requirements, using 800 (Nvidia K40) GPUs for 4 weeks (i.e. over
half a million GPU hours in total) [14, 115].

Google Brain’s famous NASNet by Zoph et al. was also built using an RL Search
Algorithm [115]. NASNet mainly differs from the initial NAS model by Zoph & Le
in that the controller uses Proximal Policy Optimization [134] instead of
REINFORCE Policy Gradient, and the Search Space is Cell-Based rather than
Layer-Wise. More impressively, NASNet obtained an even better error-rate (as
low as 2.40%) on the same CIFAR-10 benchmark, with the copious amount of
computational requirements.

####  III-B3 Neuroevolution/Evolvable-NAS

Two of the most prominent Search Algorithms for NAS are Neuroevolution (NE)
and Evolvable Neural Architecture Search (ENAS); while both methodologies are
based on Evolutionary Algorithms (EAs), NE aims to optimize both the structure
and the weights of a network, whereas ENAS is only concerned with the
architecture [13]. NE and ENAS draw inspiration from biological evolution,
where an architecture corresponds to an individual in a population that could
breed (other architectures with similar features) if it is well-performing, or
get removed from the population otherwise. Simulating this “survival of the
fittest” approach in-silico has continued to show promising results for over 3
decades [135, 124, 136, 122].

input _: A dataset D, number of generations T, number of individuals in a
generation N, and the probabilities of mutation and crossover $P_{m}$ and
$P_{c}$_ initialization _: Generate a set of randomized individuals (neural
architectures) $\\{\mathbb{M}_{0},n\\}_{n=1}^{N}$ and evaluate their fitness_
for  _t = 1, 2, …, T_ do

Selection _: select a new generation
$\\{\mathbb{M}^{{}^{\prime}}_{t,n}\\}_{n=1}^{N}$ based on the fitness scores
of $\\{\mathbb{M}_{t-1,n}\\}_{n=1}^{N}$_ Crossover _: for each pair
$[\mathbb{M}_{t,2n-1},\mathbb{M}_{t,2n}]^{[N/2]}_{n=1}$, perform crossover
with a probability of $P_{c}$_ Mutation _: for each non-crossover topologies
$\\{\mathbb{M}_{t,n}\\}_{n=1}^{N}$, perform mutation with a probability of
$P_{m}$_ Fitness evaluation _: compute the fitness for all generated neural
architectures $\\{\mathbb{M}_{t,n}\\}^{N}_{n=1}$_

end for

Output _the last generation of ANN architectures
$[\mathbb{M}_{T,n}]^{N}_{n=1}$ with their fitness scores (typically accuracy /
error)_

Algorithm 1 A generic Evolvable NAS pseudocode (adapted from [137])

Evolutionary NAS, although a relatively newer area than Neuroevolution, has
gained increasing momentum since 2017 [136, 123, 137, 138, 121, 122]. ENAS
models typically use GAs to evolve and optimize the neural structure, then
train the network using Stochastic Gradient Descent (SGD) [13]. AmoebaNet,
proposed by Real et al. in 2019 [122], applies the Tournament Selection [139]
method, where the best candidate is selected out of a random sample set $S$
and the candidate’s mutated offspring gets put back into the population
(sample size $|S|=1$ would turn this into a Random Search algorithm).
AmoebaNet also implements a selection approach called aging evolution that
favors younger genotypes, which puts the exploration of the search space ahead
of the exploitation (i.e prevents the search algorithm from narrowing down on
networks that perform well early). Tournament Selection has also been used in
a number of other successful ENAS models [136, 123, 140]. On the other hand,
Elsken et al. employed a multi-objective Pareto front optimization approach to
sample parents inversely proportional to their density [121].

Although RL Search Algorithms tend to be the focus of attention in the NAS
research community, a survey conducted by Real et al. on RL, NE/ENAS, and RS
has concluded that Reinforcement Learning and evolutionary approaches perform
similarly in terms of final test accuracies. Moreover, Neuroevolution and
Evolutionary NAS had a slightly better anytime-performance and were more
inclined to find smaller, more efficient models [122]. These findings,
however, do not factor in the optimization convergence profiles. Conversely to
RL and EAs, Gradient Optimization (GO) excels at fast convergence over smaller
periods of search time; gradient calculations provide a robust convergence
trajectory every iteration whereas discrete optimization requires prolonged
training to provide a sufficiently reliable approximation.

####  III-B4 Gradient Optimization

Following the concepts introduced by Shin et al. [41] and Liu et al. [127],
transforming a search space into a continuous space was popularized and thus
Gradient Optimization became a viable algorithm for NAS. Widely considered the
basis of differentiable NAS, Differentiable Architecture Search (DARTS) uses a
continuous Cell-Based Space search space, where each cell represents a node in
a sequentially-built DAG, which can then be simultaneously relaxed and
optimized using GO [127]. The differentiable nature of the Continuous Search
Space also enables DARTS to jointly optimize the architecture and the weights
of the network using gradient descent.

On the other hand, Dong & Yang proposed to forgo the bi-level optimization
used in DARTS and instead prioritized reducing the search cost through
sequential 2-stage optimization [141]. In other words, while DARTS optimizes
the weights and topology in the form of a nested loop (which unnecessarily
repeats the relaxation process of the search space DAG), GDAS performs each
optimization process separately in a consecutive manner, which reduces the
complexity of the search. Additionally, GDAS incrementally relaxes each edge
of the search space graph rather than the entire structure simultaneously,
further breaking down the optimization process into smaller and more
manageable problems. Consequently, this approach significantly reduces the
overall search efficiency and is able to find competitively-performing
architectures in just 4 GPU hours, making it one of the most efficient NAS
methods in the state-of-the-art.

####  III-B5 Other Search Algorithms

In addition to the aforementioned mainstream optimization algorithms for NAS,
Surrogate Model-Based Optimization (SMBO) (often interchangeable with Bayesian
Optimization (BO) [142, 118]) has been consistently successful in many areas
for almost a decade [143, 144, 107, 145, 146]. Based on Bayes Theorem, BO
works by building a probabilistic model (called the surrogate model) of the
objective/cost function. A selection/acquisition function can then be used to
sample data points from the posterior, which would iteratively approximate the
solution surface. There are numerous surrogate model and acquisition function
definitions; however, most BO models use Gaussian Processes (GPs) as surrogate
models and Expected Improvement Criterion (modeled below) as selection
functions [142].

$E.I_{y^{*}}(x)=\int_{-\infty}^{\infty}max(y-y^{*},0)\,p(y|x)\,dy$ (3)

the Expected Improvement (EI) function (Eq. 3) is built over the surrogate
model $p(y|x)$, where $y$ is the objective function, $y^{*}$ is the maximum
observed value of $y$, and $x$ is the data-point (the hyperparameter in the
case of hyperparameter optimization, or the network in case of NAS). The
integral (or, in practice, a discrete and bounded variation of which) yields
the approximate solution space surface, where the maximum observed value
$y^{*}$ represents the global maximum, or the optimal solution.

Closely related to NE/ENAS, Swarm Intelligence (SI) optimization approaches
also have been applied to NAS and yielded impressive results. Niu et al.
proposed a Particle Swarm Optimization (PSO)-based NAS model that has achieved
state-of-the-art accuracy (95.38% on CIFAR10) in just 0.2 GPU days whilst
maintaining an incredibly small model size [147]. Other SI algorithms, such as
Ant Colony Optimization [148, 149, 150], have also been briefly explored in
the NAS context. Further research might be needed, however, to cover the
unexplored SI approaches (Microbial Intelligence, Fish School Search, Glowworm
Algorithm, etc.) for NAS.

###  III-C Evaluation Strategies

Search Algorithm |  Reference / Model |  Year |  Search Space |  Evaluation Strategy |  GPU Usage (days)11footnotemark: 1 |  Params (millions) |  CIFAR10 Error (%)
---|---|---|---|---|---|---|---
|  Luo et al. (NAONet Random-WS) [151] |  2018 |  Hierarchical |  OS |  0.25 |  3.90 |  3.92
|  Brock et al. (SMASHv2) [128] |  2018 |  MBR |  OS |  16 |  4.60 |  4.03
|  Li & Talwalkar (RandomNAS) [132] |  2020 |  Cell-Based |  OS |  2.7 |  4.30 |  2.85
RS |  Zhang et al. (Random-NSAS) [152] |  2020 |  Cell-Based |  OS |  0.7 |  3.08 |  2.64
|  Liu et al. (DARTS (second order)) [127] |  2018 |  Continuous Cell-Based |  OS |  4 |  3.30 |  2.76
|  Cai et al. (ProxylessNAS-G) [120] |  2018 |  Continuous Cell-Based |  OS |  8.33 |  5.70 |  2.06
|  Dong & Yang (GDAS) [141] |  2019 |  Continuous Cell-Based |  OS |  0.17 |  2.50 |  2.82
GO |  Zhou et al. (BayesNAS) [145] |  2019 |  Continuous Hierarchical |  OS |  0.1 |  3.40 |  2.41
|  Liu et al. (PNAS) [118] |  2018 |  Cell-Based |  LFE |  - |  3.20 |  3.41
SMBO |  Kandasmy et al. (NASBOT) [153] |  2018 |  Layer-Wise |  LFE |  1.6 |  - |  12.09
|  Baker et al. (MetaQNN) [113] |  2016 |  Layer-Wise |  Full Training |  90 $\pm$ 10 |  11.20 |  6.92
|  Zoph & Le (NAS) [14] |  2017 |  Layer-Wise |  Full Training |  22,400 |  37.40 |  3.65
|  Zoph et al. (NASNet-A) [115] |  2018 |  Cell-Based |  Full Training |  2,000 |  3.30 |  3.41
|  Cai et al. (EAS) [114] |  2018 |  Hierarchical |  Weight Inheritance |  17.5 $\pm$ 7.5 |  23.40 |  4.23
|  Pham et al. (Efficient NAS) [154] |  2018 |  Cell-Based |  OS |  0.45 |  4.60 |  2.89
RL |  Ding et al. (BNAS) [155] |  2021 |  Cell-Based |  OS |  0.19 |  4.80 |  2.88
|  Stanley & Miikkulainen (NEAT) [12] |  2002 |  Neuronal |  - |  - |  - |  -
|  Stanley et al. (HyperNEAT) [124] |  2009 |  Neuronal |  - |  - |  - |  -
|  Real et al. (Large-Scale Evolution) [136] |  2017 |  Layer-Wise |  Weight Inheritance |  2,600 [66] |  5.40 |  5.40
|  Liu et al. (Hierarchical EAS-Evolution) [123] |  2018 |  Hierarchical |  Full Training |  300 |  15.70 |  3.75
|  Real et al. (AmoebaNET-A) [122] |  2019 |  Cell-Based |  Full Training |  3,150 |  3.20 |  3.34
|  Elsken et al. (LEMONADE) [121] |  2019 |  Cell-Based |  Weight Inheritance |  40 $\pm$ 16 |  3.40 |  3.60
|  Lu et al. (NSGA-Net) [156] |  2019 |  Cell-Based |  LFE |  4 |  3.30 |  2.75
|  Guo et al. (SPOS) [157] |  2020 |  Cell-Based |  OS |  - |  - |  -
NE/ENAS |  Yang et al. (CARS) [158] |  2020 |  Cell-Based |  OS |  0.4 |  3.60 |  2.62
|  Niu et al. (PNAS) [147] |  2019 |  Cell-Based |  Weight Inheritance |  0.2 |  2.44 |  4.62
|  Byla & Pang (DeepSwarm) [148] |  2019 |  Layer-Wise |  LFE |  - |  - |  11.31
|  Mo et al. (SA-NAS-c) [159] |  2021 |  Cell-Based |  LFE |  0.1 |  3.20 |  2.53
MO |  Shahawy & Benkhelifa (HiveNAS) [160] |  2022 |  Layer-Wise |  LFE |  0.3 |  1.39 |  8.90

TABLE II: Performance comparison for NAS models based on their Search Space,
Search Algorithm, and Evaluation Strategy

In order to obtain feedback to guide the Search Algorithm and further optimize
the sampled solutions, an Evaluation Strategy is need to estimate the samples’
performance. The choice of Evaluation Strategy does not tremendously affect
the search process, rather it can have a great impact on the computational
demands of the model as the evaluation process is performed at every iteration
[161].

####  III-C1 Full Training

Considered the simplest form of performance estimation, Full Training (often
termed “Training from Scratch”) simply trains every candidate network until
convergence and then estimates the models’ accuracy and/or other metrics. This
approach, although straightforward and provides the most accurate feedback, is
the most computationally demanding Evaluation Strategy; every candidate
network has to undergo a full training and testing phase to produce a single
feedback data sample.

Models using this naïve approach of evaluation normally require a large number
of GPUs running in parallel, leading to excessively high resource demands (in
the order of thousands of GPU days) [14, 115, 122, 162, 163]. These unfeasible
requirements prompted the development of new evaluation methodologies that
reduce the performance estimation time.

####  III-C2 Lower Fidelity Estimation

One way to effectively reduce the evaluation time is by training each
candidate architecture on (i) just a subset of the data [164, 165], (ii) for a
fewer number of epochs (early-stop) [115], (iii) using a down-scaled version
of the dataset (lower resolution images, etc.) [116], and/or (iv) using a
compressed version of the candidate model [115, 122, 166]. This method is
commonly referred to as Lower Fidelity Estimation (LFE; sometimes denoted as
Proxy Task Metrics).

Lower Fidelity Estimation, although significantly faster and cheaper than the
traditional Full Training approach, has its downsides as well. Compressing the
evaluation training set/model presents an obvious risk of bias in the
candidates’ performance estimation process [167]. The resulting bias, however,
can be negligible given that the Search Algorithms can typically be guided
using the relative performance rankings of the candidate networks.
Nevertheless, if the difference between the biased and fully-trained metrics
is too large, the candidates’ rankings might become relatively inconclusive
and a multi-fidelity approach might be needed [168, 169, 170].

####  III-C3 Weight Inheritance

Another way to circumvent high computational demands is to rely on Weight
Inheritance (WI) from a parent model (often denoted as Network Morphism), an
approach similar to Net2Net’s knowledge transfer [171]. By initializing each
candidate network to a relevant function, or by simply using the Full Training
approach once and then only fine-tuning the weights every subsequent
generation, the evaluation time can be cut down significantly.

Furthermore, because most Search Algorithms (specifically ENAS-based models
[13]) do not tremendously disrupt the architectures of new individuals, Weight
Inheritance can be applied to the unchanged areas of the topology and most of
the parameters can be swiftly passed down across generations without requiring
retraining from scratch. This method has been proven to efficiently evaluate
candidate networks by numerous NAS models [121, 114, 120, 172].

####  III-C4 Performance Predictors

An alternative Evaluation Strategy is through the use of Performance
Predictors. A particularly popular performance prediction approach is Learning
Curve Extrapolation (LCE), which initializes the candidate networks through
partial training (early-stopping) and then uses a surrogate model to estimate
the generalization of the final learning curve [173]. Some studies consider
Learning Curve Extrapolation to be a subset of Lower Fidelity Estimation given
its early-stopping nature [174]; however, an argument could be posed that the
use of fewer epochs to train candidate networks is only one step in this
Evaluation Strategy rather than its whole premise, hence Learning Curve
Extrapolation is typically considered its own distinct methodology [74, 175].

While many models implementing Learning Curve Extrapolation have achieved
state-of-the-art performance with impressively low computational requirements
[176, 177, 173], the surrogate model often requires extensive training (both
weight and hyperparameter optimization) to yield satisfactory extrapolation
performance. Nevertheless, Ru et al. have shown that sufficient surrogate
model performance can be achieved under a fixed-hyperparameter framework
[175].

####  III-C5 One-Shot Models

A particularly unique approach to search for neural architectures is One-Shot
(OS) models, which heavily rely on weight-sharing (not to be confused with the
One-Shot Learning paradigm that aims to train a model using one input sample).
The premise of OS methods is to build a single over-parameterized network (a
large DAG, often defined as the super-network/supernet) whose sub-graphs are
individual candidate models, sharing their weights on common edges. The
training process for one super-network is only slightly more complex than that
of any individual candidate in the search space [76]. This approach is often
categorized as a distinct “Search Strategy” rather than any individual
component from Fig. 3 [178]. However, given that the primary aim of OS NAS is
to reduce search cost through a single training phase, we classify the One-
Shot NAS approach as an Evaluation Strategy to conform with the taxonomy
adopted by the research community.

Brock et al. proposed one of the earliest OS models, SMASH [128]. SMASH
operates by generating pseudo-random weights through a surrogate “HyperNet”,
which (although randomly-generated) are sufficiently optimal to provide a
relative measure for the candidate sub-graphs. Sampling hundreds of
architectures can be done in under 1 GPU day due to the concurrent
architecture/weights search, which (compared to other Search
Algorithms/Evaluation Strategies) is significantly fast.

In the context of adaptive learning, one apparent limitation of OS models is
that the super-network is static and thus constrains the Search Space to its
sub-graphs. Furthermore, One-Shot models are (i) immensely restricted by the
GPU’s capacity given that the supernet normally needs to be loaded into the
memory during the search process, and (ii) they inherently do not enable
individual sub-graph optimization due to weight-sharing.

Notwithstanding the achievements accomplished in the NAS domain, frameworks
generally assume a non-incremental experimental setup where all data is
present prior to architecture generation, which limits their use to static
environments. In the next section, we explore how NAS can be used in lifelong
learning settings and formalize autonomous adaptiveness for ANNs.

##  IV Continually-Adaptive Neural Networks

Humans have evolved to be capable of general learning and skill-refinement;
the essence of our intelligence is captured by our ability to continuously
assimilate new knowledge and adapt to the varying stimuli we are exposed to
(i.e Classical Conditioning) [179, 180]. Most modern ANNs are models of what
is referred to as dedicated intelligence [181], or domain-specific
intelligence, which is a product of the particularly restrictive (albeit
efficient) single-task methodologies commonly used today [17].

Human-like intelligence has long been the ultimate goal of research in AI
[182]. Although the definition of “human intelligence” is largely tacit, one
of its most defining features is adaptive self-modification [183].

###  IV-A Definitions

In the context of Artificial Neural Networks, the existing definitions of the
term “adaptive” are inconsistent. Most early studies described weight-
modification as a type of adaptiveness (implying that all neural networks are
fundamentally adaptive) [184, 185], while more recent works generally confined
the definition to specific areas (i.e. domain-adaptive [186], mode-adaptive
[187], etc.).

Derived from Hebb’s principle on Synaptic Plasticity, the term “Plastic”
Artificial Neural Networks (PANNs) has been used to describe adaptiveness on a
cellular/synaptic level [188]. These self-managed autonomous agents, however,
are exclusively researched in Unsupervised Learning settings using
Neuroevolution [69, 17]. To break out of the restricted scope of
Neuroevolution and address continual autonomy in Supervised Learning
environments, we propose and formalize the Continually-Adaptive Neural
Networks (CANNs) paradigm. We define an ANN as continually-adaptive if it is:

  1. 1.

fully-autonomous throughout its life-cycle,

  2. 2.

inherently continual (i.e. has CL attributes, regardless of its Incremental
Learning setting), and

  3. 3.

capable of handling and adapting to an infinite stream of incremental input
(with allowed degradation of performance over time; see Section IV-C-4).

Figure 6: Proposed taxonomy for Continually-Adaptive Neural Networks

Although there are numerous additional components that could be considered
adaptive (e.g. learning rule plasticity), Fig. 6 defines the baseline,
mandatory elements that would ensure end-to-end adaptiveness in a typical
supervised learning ANN life-cycle.

Moreover, it is important to highlight that – according to Criterion 2 – a
CANN should not simply be an iterative AutoML pipeline. While there exists
several approaches that can automate pre-training data preparation and model
generation, it is desirable for every component to have Continual Learning
attributes; the learning process should not be iterated with statistical
independence for every task encountered, but rather enable Forward and
Backward Transfer wherever possible.

###  IV-B CANN Components

A CANN framework comprises of 2 essential components: Data Automation and
Model Plasticity.

The first stage in a standard Machine Learning pipeline is Data Preparation,
which includes all procedures required before the training stage. These
procedures include Data Acquisition, Data Pre-processing, and Feature
Engineering (as shown in Fig. 7) [72, 189, 73].

####  IV-B1 Data Acquisition

In particular real world applications, a continuous source of data (e.g. a
sensor, live camera, etc.) is attached to the model, rendering the Data
Acquisition step unnecessary. However, in other scenarios, the data-source is
either insufficient (provides a limited amount of data) or simply does not
exist. Considering the importance of the dataset’s size and its impact on a
model’s performance, a number of approaches have emerged to extend limited
data sources. The most common techniques to expand small data in an automated
setting are Web Search / Scrape and Data Synthesis.

With the increasing availability of public datasets, web-scraping [190, 191,
192] and dedicated
APIs222htttps://www.datasearch.elsevier.com333https://datasetsearch.research.google.com/
[193] have become viable solutions for automated Data Acquisition. The main
concerns with such Data Search approaches are:

  * •

The legality and ethics of web-scraping [194].

  * •

Search results are likely to contain irrelevant data.

  * •

Unpredictable data quality; without a supplementary data-quality verification
technique, the acquired data is unreliable.

Data Synthesis, in contrast, offers a more controllable Data Acquisition
environment that overcomes all the listed drawbacks posed by Web
Search/Scrape. The fast-growing research on generative models and data
simulators has shown that synthetic data can be an impressive alternative when
real data is unattainable. The mesmerizing realism produced by Generative
Adversarial Networks (GANs) [195] also proves that synthetic data can be
almost indistinguishable from their real counterpart whilst providing granular
control of the results [196]. Moreover, tools like OpenAI Gym, Unreal Engine,
and even the Grand Theft Auto game have been successfully used to simulate
entire scenes for image classification and segmentation [197, 198]. The
apparent limitation of Data Synthesis is that the process (in an automated
context) requires an initial dataset to establish a baseline distribution for
the synthesizer to match.

Figure 7: Continual Data Automation flowchart

####  IV-B2 Data Pre-processing

Whether the data is directly fed to the framework through a continuous source
or externally acquired, Data Cleaning is inevitable. The cleaning process
includes removing noise/outliers, handling missing data (commonly denoted as
NaNs), and eliminating inconsistencies [199]. Although the Data Pre-processing
stage is heavily dependent on the domain of the problem (image, text, audio
signals, etc.), some data cleaning algorithms, such as BoostClean [200] and
AlphaClean [201], are domain-agnostic and therefore enable generic automation.
Moreover, Continuous Data Cleaning frameworks that leverage the user’s
feedback on cleaning preferences have been briefly explored [202].

Amongst the issues that commonly arise during Data Preparation are class
imbalance (uneven data per label), and lack of variability/over-fitting. By
augmenting the data (i.e using affine transformations on image datasets or
pitch-scaling on audio signals; we refer the reader to the survey by He et al.
[73] for a Data Augmentation taxonomy), new data points can be generated to
help balance the dataset and increase the generalization ability of the model.
A byproduct of Data Augmentation is dataset extension and so it can also be
regarded as a Data Acquisition approach. However, the consensus is that
augmentation is primarily used for its regularization qualities and is thus
considered a part of the Data Pre-Processing stage [203, 73].

####  IV-B3 Feature Engineering

Reference / Model |  NAS Optimizer |  Search Space |  Eval. Strategy | Continual Learning Type |  Benchmark |  Error (avg. %)
---|---|---|---|---|---|---
Li et al. 2019 (Learn-to-Grow) [204] |  GO |  Continuous |  OS |  Task-Incremental |  Prior-Focused & Parameter Isolation |  ImageNet + CIFAR100 + \+ 8 others (avg) |  30.16, 20.41, 22.32
Huang et al. 2019 (CNAS) [205] |  RL |  Layer-Wise |  WI |  Class-Incremental |  Episodic Rehearsal |  CIFAR100 (k=10) |  36.50
Pasunuru & Bansal 2019 (CAS) [206] |  RL |  Cell-Based |  LFE |  Task-Incremental |  Prior-Focused |  MSR-VTT + MSVD |  55.50, 38.30
Zhang et al. 2020 (E2NAS) [207] |  GO |  Continuous |  OS |  Task-Incremental |  Prior-Focused |  CIFAR10 + CIFAR100 + ImageNet16 |  6.11, 27.95, 34.33
Zhang et al. 2020 (REC) [208] |  RL |  Layer-Wise |  WI |  Task-Incremental |  Episodic Rehearsal & Prior-Focused |  Permuted MNIST + MNIST Variation + CIFAR100 |  4.30, 28.50, 40.30
Wu et al. 2020 (Firefly) [125] |  MO |  Neuronal, Cell-Based |  WI |  Class-Incremental |  Parameter Isolation |  CIFAR100 (k=20) |  8.97
Du et al. 2021 (ENAS-S) [209] |  EA |  Cell-Based |  LFE |  Class-Incremental |  Parameter Isolation |  CIFAR10, CIFAR100 |  10.00, 26.50
Niu et al. 2021 (AdaXpert) [210] |  RL |  Cell-Based |  WI |  Class-Incremental |  Data-Focused |  ImageNet100, ImageNet1000 |  19.26, 18.87
Zhang et al. 2021 (NSAS) [211] |  RS |  Cell-Based |  OS |  Task-Incremental |  Parameter-Isolation |  CIFAR10 + CIFAR100 + ImageNet16 |  6.45, 29.31, 57.86
Qin et al. 2021 (BNS) [212] |  RL |  Layer-Wise |  OS |  Task-Incremental |  Episodic Rehearsal |  MNIST, CIFAR10, CIFAR100 |  0.13, 8.60, 17.61
Mundt et al. 2021 (DP-NAS) [213] |  RL |  Layer-Wise |  LFE |  Task-Incremental |  Generative Rehearsal |  MNIST + FashionMNIST, MNIST + CIFAR10 |  0.21, 0.63, 23.69, 34.85
Gao et al. 2022 (CLEAS) [214] |  RL |  Neuronal |  WI |  Domain-Incremental, Class-Incremental |  Parameter Isolation |  Permuted MNIST, Rotated MNIST, CIFAR100 |  3.70, 3.00, 33.10

TABLE III: Comparison of existing Continually-Adaptive models (CANNs)

More often than not, a dataset contains redundant features that negatively
impact the convergence of the model during training. The function of any
neural layer is to essentially map an input feature vector
$\vec{x}=[x_{1},x_{2},...,x_{n}]\in\mathbb{R}^{n}$ to an output vector
$\vec{y}=[y_{1},y_{2},...,y_{m}]\in\mathbb{R}^{m}$ with a nonlinearity, which
can be modelled as:

$\hat{y}=\sum_{i=1}^{n}{x_{i}W_{i}+b_{i}}\>,$ (4)

$\vec{y}=f(\hat{y})$ (5)

Where $W=[w_{1}^{\top},w_{2}^{\top},...,w_{n}^{\top}]\in\mathbb{R}^{n\times
m}$ is the parameter matrix of the network,
$\vec{b}=[b_{1},b_{2},...,b_{m}]\in\mathbb{R}^{m}$ is the bias vector, and
$f(\cdot)$ is an element-wise nonlinearity (e.g. Tanh, Sigmoid, etc.).

A training matrix $X=[\vec{x},\vec{y}]$ with correlated features (i.e
collinearity or multi-collinearity) is linearly dependent and thus $X$ is not
a full rank matrix. Given the high-dimensional nature of deep neural networks,
where $m$ and $n$ are often in the order of thousands [117], rank-deficiency
significantly impacts the training time of the model [215]. Feature Selection
not only speeds up the convergence of the optimization process, it also
enables the use of techniques like Low-Rank Approximation that can
significantly reduce the memory-footprint of the model by representing the
weight tensor as multiple smaller tensors through low-rank factorization
[216].

Due its non-trivial and creativity-driven nature, Feature Transformation and
Construction is usually considered the most human-dependent Feature
Engineering phase [72, 73]. Constructing new features extends the dataset and
its generalization by synthesizing new features from existing ones (as opposed
to Data Augmentation, which adds new samples/rows rather than columns),
whereas Feature Transformation affects the speed of the optimization process.
Linear feature transformations, such as normalization and standardization,
boost the convergence of the training process by scaling the raw data
distribution and ensuring the features have similar magnitudes. Additionally,
the traditional use of a single learning rate in ANNs upholds the importance
of unifying the features’ scale; in a Gradient Descent training scenario, a
global learning rate applied to unnormalized data will most likely propel each
gradient/dimension at proportionally different magnitudes.

Similar to Feature Selection, Dimensionality Reduction (often called Feature
Extraction) aims to separate redundant data from effective information. The
primary distinction between the two processes is that Dimensionality Reduction
creates new compressed mappings of the original features rather filter them as
with Feature Selection. Although neural networks perform Dimensionality
Reduction inherently (e.g. pooling or convolution layers extract features from
localized regions), methods such as Principal Component Analysis (PCA) can
scale a neural network’s input and reduce the number of operations performed,
thereby minimizing the computational requirements for training [217].

Feature Engineering can be regarded as a stochastic optimization problem and
therefore a number of traditional optimization algorithms can be used to
automate the process. The limited Automated Feature Engineering frameworks
that have been proposed are mainly based on Genetic Algorithms [218, 219] and
Reinforcement Learning [220], where both approaches performed impressively
well compared to baselines (random- and manual/human-based). With the current
lack of a unified benchmark, however, automated Data Preparation frameworks
cannot be formally ranked or compared.

####  IV-B4 Neural Architecture Generation and Optimization

The crucial phase of generating a network’s topology for a given task has been
discussed in depth in Section 3. Nevertheless, the continuous maintenance of a
neural network should not be interpreted as a repeated NAS process; while
reiterating a NAS algorithm for each encountered task can indeed result in a
fully-autonomous continuous pipeline, the computational cost and time required
to repeatedly search for architectures from scratch will likely be unfeasible
for most applications. To address the expected high resource demand for
continual NAS frameworks, a few models have been recently proposed that
incorporate CL techniques within the architecture search process (e.g. finding
an optimal NAS cell that best fits all encountered tasks and facilitates few-
shot learning for future tasks [206]).

Proposed by Gao et al. in 2022, Continual Learning with Efficient Architecture
Search (CLEAS) dynamically expands the network architecture whilst preserving
previous knowledge through parameter isolation [214]. While this approach
experiences no forgetting, it scales poorly; for every encountered task, the
network has to expand with no means of compressing superfluous parameters.
Similarly, ENAS-S [209] and Learn-to-Grow [204] also experience monotonic
expansion without a memory-boundary, which makes them unsustainable for long-
term real world deployment.

Moreover, Continual NAS (CNAS) [205], REC [208], and BNS [212] all require a
dedicated memory to store subsets of training data for replay purposes. While
this memory-retention strategy offers great control over the retraining of the
models as well as the NAS process, they are not capable of handling an
infinite stream of input due to the unrestricted external memory requirement.

An alternative approach proposed by Nïu et al. (dubbed AdaXpert) dynamically
adjusts the neural architecture using a secondary model called the Neural
Architecture Adjuster (NAA), which aims to search for a new topology only when
deemed necessary (i.e the distance between task boundaries is greater than a
given threshold) [210]. Tested incrementally on ImageNet1000, one of the most
challenging vision benchmarks, this approach achieved state-of-the-art
performance in just 2.5 to 7 GPU days, proving that combining NAS and CL
optimization techniques is computationally feasible. However, in the
experiment proposed for AdaXpert, the NAA uses Wasserstein distance for every
input to compute how far the new task boundary is from the existing trained
model’s probability distribution, which shows an implicit and unmentioned
assumption that the input samples must be grouped prior to training. Given a
real world scenario where samples are interleaved, the NAA will be prompted to
adjust the network for every increment, rendering it essentially futile.

####  IV-B5 Hyperparameter Optimization

Most NAS frameworks use the same set of hyperparameters over the search
process [73]. While static hyperparameters might be viable for single-task NAS
problems, they cannot be efficiently generalized for unknown future tasks,
therefore continuous Hyperparameter Optimization is needed to maintain
efficiency across sequentially encountered tasks.

Furthermore, most NAS frameworks consider the architecture and hyperparameter
optimization processes as two separate phases [208, 73], which exponentially
increases the complexity of the framework. To overcome nesting architecture
search and hyperparameter tuning, jointly optimizing both the structure and
its hyperparameters can significantly reduce the framework’s requirements,
especially on a continual basis. A number of studies on evolutionary ANNs
suggest that hyperparameters can be endogenous features (i.e encoded as genes)
and therefore get bootstrapped into the optimization process [17].

###  IV-C Desiderata for CANNs

While the formal definition previously proposed in Section IV-A describes the
minimum qualification requirements for continual adaptiveness, there are
several desirable qualities that could assist in generalizing CANNs towards a
more general form of AI. The desiderata curated below stem from a holistic
adaptiveness perspective, as well as an individualistic CL/NAS standpoint.

  1. 1.

Domain-agnostic: The model should not rely on a priori task-boundary
definition; be capable of adapting to its surrounding environment without
exhibiting a preference to a certain domain.

  2. 2.

Abstract representation: Encoding the network should be represented with a
high level of abstraction (i.e in the case of Evolutionary Computation, the
genotype-phenotype relation should be indirect). This ensures model-
scalability and follows on the biological analogy where the human genome is
indirectly mapped to physical body [221].

  3. 3.

Dynamic architecture with bounded capacity: The network topology should grow,
shrink, or get remapped to accommodate different input complexities and
maintain efficiency. However, the architecture-search process should be
bounded by a given memory capacity to avoid overflow and facilitate deployment
in real-world applications.

  4. 4.

Graceful Forgetting: With a bounded memory capacity and a potentially
unbounded stream of data, forgetting is inevitable. It should, nevertheless,
be imposed gracefully through selective means, where the less important tasks’
performance gradually decays rather than catastrophically fails.

  5. 5.

Few-Shot Learning capability (positive FWT): Positive FWT is not often the
focus for most Continual Learning models; however, for CANNs, facilitating
Few-Shot Learning is a priority in order to reduce the large computational
costs over time.

  6. 6.

Multi-Modal input support: To further pave the way for a more generalized form
of AI, a CANN framework should ideally support knowledge-acquisition through
different modalities (e.g. image classification, text classification).

  7. 7.

Architecture Search Space inference: The effectiveness of NAS frameworks is
almost exclusively influenced by the Search Space design [222], which is
typically a human-dependent process. Dynamic Search Space design for NAS is a
favorable quality for CANNs to facilitate domain-agnostic adaptiveness.

  8. 8.

No explicit data storage for previous tasks: Storing subsets of examples from
previous tasks usually entails linearly increasing memory-consumption.
Parameter-Constraint approaches, where the model’s “memory” is implicitly
stored through the training process, are therefore considered a more favorable
memory-retention technique. Additionally, the globally growing privacy
concerns also dictate less data-storage-based methodologies.

  9. 9.

Learning rule plasticity: The training process for ANNs greatly relies on
predefined learning rules (or weight-updating equations) that determine the
changes to the network parameters. Jointly optimizing the learning rule with
the other network parameters (as suggested by [17, 221]) could effectively cut
down the training cost.

Although these features do not directly contribute to end-to-end automation
and continuity, they aim to generalize CANNs and facilitate their scalability
for real-world deployment.

##  V Future Directions

A palpable concern regarding CANNs is the potentially staggering computational
requirements; pipelining multiple automated optimization approaches
simultaneously with a possibly expanding network can be computationally
expensive. Nevertheless, several models (Table III) have showcased the
feasibility of CANNs in all Incremental Learning settings. Moreover, as per
criterion 2 (IV-A-2), Continual Learning should prospectively be incorporated
within the NAS process to enable knowledge transfer across learning
increments, further lowering the optimization costs. This fundamental
inclusion of CL into the learning process has never been explored.

While AutoML frameworks (NAS specifically) and Continual Learning models have
implicitly agreed upon benchmarks in the literature, the intersection between
both fields does not (as shown in Table III). Some works use the average
accuracy over all tasks to rank their model [204, 210]. However, that metric
diminishes as the number of tasks approaches infinity. Consider a model with a
bounded memory capacity for $\mathbb{C}$ tasks and an unbounded incremental
input $x\>,\;s.t.\;|x|=+\infty$, the running average of accuracies will reach
an inflection point at $x_{\mathbb{C}}$ where the inevitable forgetting will
set in and the average value is bound to diminish. Furthermore, in the context
of a system that enables an infinite stream of input, the model should not be
ranked according to the “final” model metrics; by definition, the framework
has no final state and intermediate results might not reflect the framework’s
true performance. Additionally, the disjoint incremental learning settings and
benchmarks cannot accurately estimate the models’ adaptiveness and
performance. Hence, a robust ranking approach for CANNs (potentially based on
the rate of change of metrics rather than instantaneous values) needs to be
formalized.

Furthermore, inconsistent with their state-of-the-art performance,
Metaheurstic Optimization for NAS is often overlooked in comprehensive surveys
and reviews [74, 66]. Scarce coverage of such approaches limits the
exploration of the optimization family, which has continued to show great
potential in multiple areas and inherently fits the parallel exploration
nature of NAS [147, 149, 159]. Some MO approaches, such as Microbial
Intelligence and Glowworm Algorithm, have never been explored in the context
of Neural Architecture Search, making them prospective future research
directions.

Another gap in the NAS domain is the reliance on predefined search spaces. Yu
et al. have concluded that the effectiveness of NAS frameworks is primarily
attributed to how the Search Space is constrained [222]; NAS models can
achieve state-of-the-art performance with minimal cost if they are explicitly
bound to a subset of the Search Space where optimal solutions exist. In the
CANNs domain, however, a priori knowledge should not be exploited (Section
IV-C-1), and thus domain-agnostic Search Space-inference methods are
necessary.

A relevant research area that is adjacent to CANNs is Social Learning
(sometimes dubbed collaborative learning). Human intelligence could not have
evolved through one individual in a single life-time; similarly, it is
unlikely that a complex set of mechanisms could evolve or be learned in a
relatively short period of time without knowledge-sharing across multiple
artificial models. Recent works on ML started exploring the concept of Social
Learning, where a ML model would share information with other models in the
same domain to collectively boost performances [181, 17]. Social Learning can
be complementary to CANNs, especially given their lifelong learning nature.

Although unsupervised self-managing ANNs have been briefly explored in the
literature [69, 17], they are exclusively researched from an
evolutionary/synaptic plasticity perspective. The unsupervised counterparts of
NAS and CL (dubbed Unsupervised NAS (UnNAS) [223] and Unsupervised Continual
Learning (UCL) [224]) have both shown promising potential individually, but
their intersection has not been defined or reviewed.

One of the primary bottlenecks for full autonomy in CANNs is the need for
manual definitions of task boundaries. The biological counterpart of lifelong
learning is contingent on spontaneous generation of goals, which are
fundamentally driven by intrinsic primal disposition to survival and
procreation [9]. Modelling these goal-seeking neural networks [225] is an open
challenge in AI that should be investigated further to enable true autonomy.

Notwithstanding the exciting prospect of generalized AI, there are drastic
risks associated with such forms of continual automation. Unmonitored data
collection introduces the possibility of accidental dataset poisoning, where
unwanted samples are fed to the model. In certain sensitive domains (UAVs,
medical applications, industrial IoT devices, etc.), data-integrity must be
verified and input data should possibly be restricted to certain distribution
boundaries as well. Safeguarding CANNs against potential task deviations would
further pave the way for safe AI.

##  VI Conclusion

In this study, we propose a dynamic paradigm that entails full-automation and
continual-adaptiveness in neural networks. Despite the existence of several
continually-adaptive models (as per Table III), this intersection between NAS
and Continual Learning has never been formalized or reviewed in depth.

Through self-development and lifelong plasticity, CANNs aim to overcome
numerous limitations posed by the human factor and result in more optimal
models. In doing so, Continually-Adaptive Neural Networks could potentially
introduce a new spectrum of applications where the functionality of the model
is pliable, even after the deployment phase when remote access is often
limited or unavailable.

Additionally, CANNs take a step towards generalizing intelligence; although
the definition of “intelligence” has long eluded psychologist and
neuroscientists, it has been repeatedly defined as ”goal-directed adaptive
behavior” and “adaptive self-modification” rather than mere associative
inference [226, 183]. Narrow AI has indeed led to amazing accomplishments,
however, there is an undeniable impending bottleneck to the capabilities that
can be achieved with traditional methodologies. Continual adaptability and
automation, on the other hand, minimize the static nature of Narrow AI and
provide an essential basis for General AI.

The proposed categorization schemes and analysis of methodologies aim to
propel researches towards a less dedicated form of intelligence and facilitate
more robust models that are sensitive to their surrounding environments.

## References

  * [1] A. J. Otoole, P. J. Phillips, F. Jiang, J. Ayyad, N. Penard, and H. Abdi, “Face recognition algorithms surpass humans matching faces over changes in illumination,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 29, no. 9, p. 1642–1646, 2007.
  * [2] D. Hassabis, “Artificial intelligence: Chess match of the century,” _Nature_ , vol. 544, no. 7651, p. 413–414, 2017.
  * [3] K. Darlington, “How good are ai reading comprehension systems?” _IQ: The RIM Quarterly_ , vol. 37, no. 4, pp. 40–43, 2021.
  * [4] M. Negrão and P. Domingues, “Speechtotext: An open-source software for automatic detection and transcription of voice recordings in digital forensics,” _Forensic Science International: Digital Investigation_ , vol. 38, p. 301223, 2021.
  * [5] Y. Sun, D. Liang, X. Wang, and X. Tang, “Deepid3: Face recognition with very deep neural networks,” _arXiv preprint arXiv:1502.00873_ , 2015.
  * [6] Q. Yu, Y. Yang, F. Liu, Y.-Z. Song, T. Xiang, and T. M. Hospedales, “Sketch-a-net: A deep neural network that beats humans,” _International Journal of Computer Vision_ , vol. 122, no. 3, p. 411–425, 2016.
  * [7] H. Y. Khaw, F. C. Soon, J. H. Chuah, and C.-O. Chow, “Image noise types recognition using convolutional neural network with principal components analysis,” _IET Image Processing_ , vol. 11, no. 12, pp. 1238–1245, 2017\.
  * [8] A. Blair and A. Saffidine, “Ai surpasses humans at six-player poker,” _Science_ , vol. 365, no. 6456, p. 864–865, 2019.
  * [9] A. Gopnik, A. N. Meltzoff, and P. K. Kuhl, _The scientist in the crib: Minds, brains, and how children learn._ William Morrow & Co, 1999.
  * [10] A. M. Zador, “A critique of pure learning and what artificial neural networks can learn from animal brains,” _Nature Communications_ , vol. 10, no. 1, 2019\.
  * [11] S. Seung, _Connectome: how the brains wiring makes us who we are_. Penguin, 2013.
  * [12] K. O. Stanley and R. Miikkulainen, “Evolving neural networks through augmenting topologies,” _Evolutionary computation_ , vol. 10, no. 2, pp. 99–127, 2002.
  * [13] Y. Liu, Y. Sun, B. Xue, M. Zhang, G. G. Yen, and K. C. Tan, “A survey on evolutionary neural architecture search,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–21, 2021.
  * [14] B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” in _ICLR_ , 2017.
  * [15] M. Mccloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” _Psychology of Learning and Motivation_ , p. 109–165, 1989.
  * [16] J. L. Mcclelland, B. L. Mcnaughton, and R. C. Oreilly, “Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory.” _Psychological Review_ , vol. 102, no. 3, p. 419–457, 1995.
  * [17] A. Soltoggio, K. O. Stanley, and S. Risi, “Born to learn: The inspiration, progress, and future of evolved plastic artificial neural networks,” _Neural Networks_ , vol. 108, p. 48–67, 2018.
  * [18] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” _Neural Networks_ , vol. 113, p. 54–71, 2019.
  * [19] G. Anthes, “Lifelong learning in artificial neural networks,” _Communications of the ACM_ , vol. 62, no. 6, p. 13–15, 2019.
  * [20] M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “A continual learning survey: Defying forgetting in classification tasks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , p. 1–1, 2021.
  * [21] J. Zhang, J. Zhang, S. Ghosh, D. Li, S. Tasci, L. Heck, H. Zhang, and C.-C. J. Kuo, “Class-incremental learning via deep model consolidation,” _2020 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2020.
  * [22] Y. Luo, L. Yin, W. Bai, and K. Mao, “An appraisal of incremental learning methods,” _Entropy_ , vol. 22, no. 11, p. 1190, 2020.
  * [23] G. M. van de Ven, T. Tuytelaars, and A. S. Tolias, “Three types of incremental learning,” _Nature Machine Intelligence_ , pp. 1–13, 2022.
  * [24] A. Robins, “Sequential learning in neural networks: A review and a discussion of pseudorehearsal based methods,” _Intelligent Data Analysis_ , vol. 8, no. 3, p. 301–322, 2004.
  * [25] J. Zevin and M. Seidenberg, “Simulating consistency effects and individual differences in nonword naming: A comparison of current models,” _Journal of Memory and Language_ , vol. 54, no. 2, p. 145–160, 2006.
  * [26] J. Chen, Y. C. Leong, C. J. Honey, C. H. Yong, K. A. Norman, and U. Hasson, “Shared memories reveal shared structure in neural activity across individuals,” _Nature Neuroscience_ , vol. 20, no. 1, p. 115–125, 2016\.
  * [27] A. Cleeremans, “The radical plasticity thesis: How the brain learns to be conscious,” _Frontiers in Psychology_ , vol. 2, 2011.
  * [28] M. S. Bennett, “What behavioral abilities emerged at key milestones in human brain evolution? 13 hypotheses on the 600-million-year phylogenetic history of human intelligence,” _Frontiers in Psychology_ , vol. 12, 2021.
  * [29] F. L. Coolidge, “The evolution of learning and memory systems,” _Evolutionary Neuropsychology_ , p. 27–53, 2020.
  * [30] T. Takeuchi, A. J. Duszkiewicz, and R. G. M. Morris, “The synaptic plasticity and memory hypothesis: encoding, storage and persistence,” _Philosophical Transactions of the Royal Society B: Biological Sciences_ , vol. 369, no. 1633, p. 20130288, 2014.
  * [31] D. E. Feldman, “Spike timing–dependent plasticity,” _Neural Circuit and Cognitive Development_ , p. 127–141, 2020.
  * [32] D. O. Hebb, _The organization of behavior: A neuropsychological theory_. Wiley, Jun. 1949.
  * [33] M. A. Lynch, “Long-term potentiation and memory,” _Physiological Reviews_ , vol. 84, no. 1, p. 87–136, 2004.
  * [34] H. C. Dringenberg, “The history of long‐term potentiation as a memory mechanism: Controversies, confirmation, and some lessons to remember,” _Hippocampus_ , vol. 30, no. 9, p. 987–1012, 2020.
  * [35] T. Tsumoto, “Long-term depression in cerebral cortex: a possible substrate of “forgetting” that should not be forgotten,” _Neuroscience Research_ , vol. 16, no. 4, p. 263–270, 1993.
  * [36] M. Escobar and B. Derrick, “Long-term potentiation and depression as putative mechanisms for memory formation,” _Neural Plasticity and Memory Frontiers in Neuroscience_ , p. 15–46, 2007.
  * [37] E. Tulving, “Organization of memory: Quo vadis,” in _The Cognitive Neurosciences_ , M. S. Gazzaniga, Ed. MIT Press, 1995, pp. 839–847.
  * [38] R. C. O’Reilly, R. Bhattacharyya, M. D. Howard, and N. Ketz, “Complementary learning systems,” _Cognitive Science_ , vol. 38, no. 6, p. 1229–1248, 2011\.
  * [39] S. Blakeman and D. Mareschal, “A complementary learning systems approach to temporal difference learning,” _Neural Networks_ , vol. 122, p. 218–230, 2020.
  * [40] R. Kemker and C. Kanan, “Fearnet: Brain-inspired model for incremental learning,” in _International Conference on Learning Representations_ , 2018\.
  * [41] R. Shin, C. Packer, and D. Song, “Differentiable neural network architecture search,” in _ICLR_ , 2018.
  * [42] J. M. Pierre, “Incremental lifelong deep learning for autonomous vehicles,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_ , 2018, pp. 3949–3954.
  * [43] Z. Sun, Y. Wang, and R. Laganière, “Online model adaptation for uav tracking with convolutional neural network,” in _2018 15th Conference on Computer and Robot Vision (CRV)_ , 2018, pp. 329–336.
  * [44] B. Wehbe, M. Hildebrandt, and F. Kirchner, “A framework for on-line learning of underwater vehicles dynamic models,” _2019 International Conference on Robotics and Automation (ICRA)_ , pp. 7969–7975, 2019.
  * [45] Y. Nose, A. Kojima, H. Kawabata, and T. Hironaka, “A study on a lane keeping system using cnn for online learning of steering control from real time images,” in _2019 34th International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC)_ , 2019, pp. 1–4.
  * [46] W. Si, T. Wei, and C. Liu, “Agen: Adaptable generative prediction networks for autonomous driving,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_ , 2019, pp. 281–286.
  * [47] M. M. Ferdaus, M. Pratama, S. G. Anavatti, and M. A. Garratt, “Online identification of a rotary wing unmanned aerial vehicle from data streams,” _Applied Soft Computing_ , vol. 76, p. 313–325, 2019.
  * [48] K. Shaheen, M. A. Hanif, O. Hasan, and M. Shafique, “Continual learning for real-world autonomous systems: Algorithms, challenges and frameworks,” _CoRR_ , vol. abs/2105.12374, 2021.
  * [49] S. Thrun and T. M. Mitchell, “Lifelong robot learning,” _Robotics and Autonomous Systems_ , vol. 15, no. 1-2, p. 25–46, 1995.
  * [50] T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, and N. Díaz-Rodríguez, “Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges,” _Information Fusion_ , vol. 58, p. 52–68, 2020.
  * [51] S. Chen and J. T. Wen, “Adaptive neural trajectory tracking control for flexible-joint robots with online learning,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 2358–2364.
  * [52] W. Zheng, H. Liu, and F. Sun, “Lifelong visual-tactile cross-modal learning for robotic material perception,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 3, pp. 1192–1203, 2021.
  * [53] A. Xie and C. Finn, “Lifelong robotic reinforcement learning by retaining experiences,” in _Advances in Neural Information Processing Systems_ , 2021\.
  * [54] G. Kahn, P. Abbeel, and S. Levine, “Badgr: An autonomous self-supervised learning-based navigation system,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 1312–1319, 2021.
  * [55] H.-C. Cheng and W.-W. Liao, “Establishing an lifelong learning environment using iot and learning analytics,” in _2012 14th International Conference on Advanced Communication Technology (ICACT)_ , 2012, pp. 1178–1183.
  * [56] M. Song, K. Zhong, J. Zhang, Y. Hu, D. Liu, W. Zhang, J. Wang, and T. Li, “In-situ ai: Towards autonomous and incremental deep learning for iot systems,” in _2018 IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 2018, pp. 92–103.
  * [57] Y. Bao and W. Chen, “Il4iot: Incremental learning for internet-of-things devices,” _Lecture Notes in Computer Science_ , p. 92–107, 2019.
  * [58] A. K. Chathoth, A. Jagannatha, and S. Lee, “Federated intrusion detection for iot with heterogeneous cohort privacy,” 2021. [Online]. Available: https://arxiv.org/abs/2101.09878
  * [59] M. Rasouli, R. Ghosh, W. W. Lee, N. V. Thakor, and S. Kukreja, “Stable force-myographic control of a prosthetic hand using incremental learning,” in _2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2015, pp. 4828–4831.
  * [60] A. Patra and J. A. Noble, “Hierarchical class incremental learning of anatomical structures in fetal echocardiography videos,” _IEEE journal of biomedical and health informatics_ , vol. 24, no. 4, pp. 1046–1058, 2020.
  * [61] A. Elskhawy, A. Lisowska, M. Keicher, J. Henry, P. Thomson, and N. Navab, “Continual class incremental learning for ct thoracic segmentation,” in _Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning_. Springer, 2020, pp. 106–116.
  * [62] J. Chauhan, Y. D. Kwon, P. Hui, and C. Mascolo, “Contauth,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 4, no. 4, p. 1–23, 2020.
  * [63] S. Srivastava, M. Yaqub, K. Nandakumar, Z. Ge, and D. Mahapatra, “Continual domain incremental learning for chest x-ray classification in low-resource clinical settings,” in _Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health_. Springer, 2021, pp. 226–238.
  * [64] Y. Liu, J. Wang, J. Li, S. Niu, and H. Song, “Class-incremental learning for wireless device identification in iot,” _IEEE Internet of Things Journal_ , 2021.
  * [65] J. Park, M. Kang, and B. Han, “Class-incremental learning for action recognition in videos,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 13 698–13 707.
  * [66] P. Ren, Y. Xiao, X. Chang, P.-y. Huang, Z. Li, X. Chen, and X. Wang, “A comprehensive survey of neural architecture search,” _ACM Computing Surveys_ , vol. 54, no. 4, p. 1–34, 2022.
  * [67] G. Raghavan and M. Thomson, “Neural networks grown and self-organized by noise,” 2019. [Online]. Available: https://arxiv.org/abs/1906.01039
  * [68] A. Kaplunovich and Y. Yesha, “Automatic hyperparameter optimization for arbitrary neural networks in serverless aws cloud,” _2021 12th International Conference on Information and Communication Systems (ICICS)_ , 2021\.
  * [69] J.-B. Mouret and P. Tonelli, “Artificial evolution of plastic neural networks: a few key concepts,” _Growing Adaptive Machines: combining Development and Learning in Artificial Neural Networks_ , pp. 251–261, 2014.
  * [70] T. Miconi, K. Stanley, and J. Clune, “Differentiable plasticity: training plastic neural networks with backpropagation,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 3559–3568.
  * [71] T. Nagarajah and G. Poravi, “A review on automated machine learning (automl) systems,” in _2019 IEEE 5th International Conference for Convergence in Technology (I2CT)_. IEEE, 2019, pp. 1–6.
  * [72] J. Waring, C. Lindvall, and R. Umeton, “Automated machine learning: Review of the state-of-the-art and opportunities for healthcare,” _Artificial Intelligence in Medicine_ , vol. 104, p. 101822, 2020.
  * [73] X. He, K. Zhao, and X. Chu, “Automl: A survey of the state-of-the-art,” _Knowledge-Based Systems_ , vol. 212, p. 106622, 2021.
  * [74] T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture search: A survey,” _Automated Machine Learning_ , p. 63–77, 2019.
  * [75] Y. Jaafra, J. L. Laurent, A. Deruyver, and M. S. Naceur, “Reinforcement learning for neural architecture search: A review,” _Image and Vision Computing_ , vol. 89, pp. 57–66, 2019.
  * [76] M. Wistuba, A. Rawat, and T. Pedapati, “A survey on neural architecture search,” 2019. [Online]. Available: https://arxiv.org/abs/1905.01392
  * [77] S. Grossberg, “Studies of mind and brain,” _Boston Studies in the Philosophy of Science_ , 1982.
  * [78] M. Mermillod, A. Bugaiska, and P. Bonin, “The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects,” _Frontiers in Psychology_ , vol. 4, 2013.
  * [79] D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6470–6479.
  * [80] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 139–154.
  * [81] R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio, _Gradient Based Sample Selection for Online Continual Learning_. Red Hook, NY, USA: Curran Associates Inc., 2019.
  * [82] C. X. Ling and T. Bohn, “A deep learning framework for lifelong machine learning,” 2021. [Online]. Available: https://arxiv.org/abs/2105.00157
  * [83] S. Farquhar and Y. Gal, “Towards robust evaluations of continual learning,” _CoRR_ , vol. abs/1805.09733, 2018.
  * [84] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [85] J. Yoon, E. Yang, J. Lee, and S. J. Hwang, “Lifelong learning with dynamically expandable networks,” in _International Conference on Learning Representations_ , 2018.
  * [86] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne, “Experience replay for continual learning,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019\.
  * [87] T. L. Hayes, G. P. Krishnan, M. Bazhenov, H. T. Siegelmann, T. J. Sejnowski, and C. Kanan, “Replay in deep learning: Current approaches and missing biological elements,” _Neural Computation_ , vol. 33, no. 11, pp. 2908–2950, 2021.
  * [88] G. Hinton, O. Vinyals, J. Dean _et al._ , “Distilling the knowledge in a neural network,” 2015. [Online]. Available: https://arxiv.org/abs/1503.02531
  * [89] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive neural networks,” 2016\. [Online]. Available: https://arxiv.org/abs/1606.04671
  * [90] R. Aljundi, P. Chakravarty, and T. Tuytelaars, “Expert gate: Lifelong learning with a network of experts,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3366–3375.
  * [91] Z. Li and D. Hoiem, “Learning without forgetting,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 12, p. 2935–2947, 2018\.
  * [92] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska _et al._ , “Overcoming catastrophic forgetting in neural networks,” _Proceedings of the national academy of sciences_ , vol. 114, no. 13, pp. 3521–3526, 2017.
  * [93] F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 3987–3995.
  * [94] N. Kamra, U. Gupta, and Y. Liu, “Deep generative dual memory network for continual learning,” 2017. [Online]. Available: https://arxiv.org/abs/1710.10368
  * [95] S.-W. Lee, J.-H. Kim, J. Jun, J.-W. Ha, and B.-T. Zhang, “Overcoming catastrophic forgetting by incremental moment matching,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [96] H. Jung, J. Ju, M. Jung, and J. Kim, “Less-forgetful learning for domain expansion in deep neural networks,” in _AAAI_ , 2018, pp. 3358–3365.
  * [97] R. Aljundi, M. Rohrbach, and T. Tuytelaars, “Selfless sequential learning,” in _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
  * [98] A. New, M. Baker, E. Nguyen, and G. Vallabha, “Lifelong learning metrics,” _arXiv preprint arXiv:2201.08278_ , 2022.
  * [99] N. Diaz-Rodriguez, V. Lomonaco, D. Filliat, and D. Maltoni, “Don’t forget, there is more than forgetting: new metrics for continual learning,” _arXiv preprint arXiv:1810.13166_ , 2018.
  * [100] R. Kemker, M. McClure, A. Abitino, T. Hayes, and C. Kanan, “Measuring catastrophic forgetting in neural networks,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, Apr. 2018.
  * [101] T. L. Hayes, R. Kemker, N. D. Cahill, and C. Kanan, “New metrics and experimental paradigms for continual learning,” _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2018\.
  * [102] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015.
  * [103] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations_ , 2015.
  * [104] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [105] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size,” _arXiv preprint arXiv:1602.07360_ , 2016.
  * [106] I. Guyon, K. Bennett, G. Cawley, H. J. Escalante, S. Escalera, T. K. Ho, N. Macià, B. Ray, M. Saeed, A. Statnikov _et al._ , “Design of the 2015 chalearn automl challenge,” in _2015 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2015, pp. 1–8.
  * [107] H. Mendoza, A. Klein, M. Feurer, J. T. Springenberg, and F. Hutter, “Towards automatically-tuned neural networks,” in _Proceedings of the Workshop on Automatic Machine Learning_ , ser. Proceedings of Machine Learning Research, F. Hutter, L. Kotthoff, and J. Vanschoren, Eds., vol. 64. New York, New York, USA: PMLR, 24 Jun 2016, pp. 58–65.
  * [108] H. Mendoza, A. Klein, M. Feurer, J. T. Springenberg, M. Urban, M. Burkart, M. Dippel, M. Lindauer, and F. Hutter, “Towards automatically-tuned deep neural networks,” _Automated Machine Learning_ , p. 135–149, 2019.
  * [109] K. Yu, C. Sciuto, M. Jaggi, C. Musat, and M. Salzmann, “Evaluating the search phase of neural architecture search,” in _International Conference on Learning Representations_ , 2020.
  * [110] B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer, “Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 10 726–10 734.
  * [111] H. Hu, J. Langford, R. Caruana, S. Mukherjee, E. J. Horvitz, and D. Dey, “Efficient forward architecture search,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019\.
  * [112] D. Baymurzina, E. Golikov, and M. Burtsev, “A review of neural architecture search,” _Neurocomputing_ , vol. 474, pp. 82–93, 2022.
  * [113] B. Baker, O. Gupta, N. Naik, and R. Raskar, “Designing neural network architectures using reinforcement learning,” _CoRR_ , vol. abs/1611.02167, 2016.
  * [114] H. Cai, J. Yang, W. Zhang, S. Han, and Y. Yu, “Path-level network transformation for efficient architecture search,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 678–687.
  * [115] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8697–8710.
  * [116] X. Gong, S. Chang, Y. Jiang, and Z. Wang, “Autogan: Neural architecture search for generative adversarial networks,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 3224–3234.
  * [117] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [118] C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 19–34.
  * [119] Z. Zhong, J. Yan, W. Wu, J. Shao, and C.-L. Liu, “Practical block-wise neural network architecture generation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2423–2432.
  * [120] H. Cai, L. Zhu, and S. Han, “Proxylessnas: Direct neural architecture search on target task and hardware,” in _International Conference on Learning Representations_ , 2019.
  * [121] T. Elsken, J. H. Metzen, and F. Hutter, “Efficient multi-objective neural architecture search via lamarckian evolution,” in _International Conference on Learning Representations_ , 2019.
  * [122] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image classifier architecture search,” in _Proceedings of the aaai conference on artificial intelligence_ , vol. 33, no. 01, 2019, pp. 4780–4789.
  * [123] H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu, “Hierarchical representations for efficient architecture search,” in _International Conference on Learning Representations_ , 2018.
  * [124] K. O. Stanley, D. B. D’Ambrosio, and J. Gauci, “A hypercube-based encoding for evolving large-scale neural networks,” _Artificial life_ , vol. 15, no. 2, pp. 185–212, 2009.
  * [125] L. Wu, B. Liu, P. Stone, and Q. Liu, “Firefly neural architecture descent: a general approach for growing neural networks,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 22 373–22 383, 2020.
  * [126] Y. Ci, C. Lin, M. Sun, B. Chen, H. Zhang, and W. Ouyang, “Evolving search space for neural architecture search,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6659–6669.
  * [127] H. Liu, K. Simonyan, and Y. Yang, “DARTS: Differentiable architecture search,” in _International Conference on Learning Representations_ , 2019\.
  * [128] A. Brock, T. Lim, J. Ritchie, and N. Weston, “SMASH: One-shot model architecture search through hypernetworks,” in _International Conference on Learning Representations_ , 2018.
  * [129] P. Liashchynskyi and P. Liashchynskyi, “Grid search, random search, genetic algorithm: A big comparison for nas,” 2019. [Online]. Available: https://arxiv.org/abs/1912.06059
  * [130] J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization.” _Journal of machine learning research_ , vol. 13, no. 2, 2012.
  * [131] R. G. Mantovani, A. L. D. Rossi, J. Vanschoren, B. Bischl, and A. C. P. L. F. de Carvalho, “Effectiveness of random search in svm hyper-parameter tuning,” in _2015 International Joint Conference on Neural Networks (IJCNN)_ , 2015, pp. 1–8.
  * [132] L. Li and A. Talwalkar, “Random search and reproducibility for neural architecture search,” in _Uncertainty in artificial intelligence_. PMLR, 2020, pp. 367–377.
  * [133] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” _Machine learning_ , vol. 8, no. 3, pp. 229–256, 1992.
  * [134] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1707.06347
  * [135] G. F. Miller, P. M. Todd, and S. U. Hegde, “Designing neural networks using genetic algorithms.” in _ICGA_ , vol. 89, 1989, pp. 379–384.
  * [136] E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin, “Large-scale evolution of image classifiers,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 2902–2911.
  * [137] L. Xie and A. Yuille, “Genetic cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 1379–1388.
  * [138] M. Suganuma, S. Shirakawa, and T. Nagao, “A genetic programming approach to designing convolutional neural network architectures,” in _Proceedings of the genetic and evolutionary computation conference_ , 2017, pp. 497–504.
  * [139] D. E. Goldberg and K. Deb, “A comparative analysis of selection schemes used in genetic algorithms,” in _Foundations of genetic algorithms_. Elsevier, 1991, vol. 1, pp. 69–93.
  * [140] Y. Chen, G. Meng, Q. Zhang, S. Xiang, C. Huang, L. Mu, and X. Wang, “Renas: Reinforced evolutionary neural architecture search,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4787–4796.
  * [141] X. Dong and Y. Yang, “Searching for a robust neural architecture in four gpu hours,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1761–1770.
  * [142] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for hyper-parameter optimization,” _Advances in neural information processing systems_ , vol. 24, 2011.
  * [143] J. Bergstra, D. Yamins, and D. Cox, “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,” in _International conference on machine learning_. PMLR, 2013, pp. 115–123.
  * [144] B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. De Freitas, “Taking the human out of the loop: A review of bayesian optimization,” _Proceedings of the IEEE_ , vol. 104, no. 1, pp. 148–175, 2015.
  * [145] H. Zhou, M. Yang, J. Wang, and W. Pan, “Bayesnas: A bayesian approach for neural architecture search,” in _International conference on machine learning_. PMLR, 2019, pp. 7603–7613.
  * [146] C. White, W. Neiswanger, and Y. Savani, “Bananas: Bayesian optimization with neural architectures for neural architecture search,” _arXiv preprint arXiv:1910.11858_ , vol. 1, no. 2, p. 4, 2019.
  * [147] R. Niu, H. Li, Y. Zhang, and Y. Kang, “Neural architecture search based on particle swarm optimization,” in _2019 3rd International Conference on Data Science and Business Analytics (ICDSBA)_ , 2019, pp. 319–324.
  * [148] E. Byla and W. Pang, “Deepswarm: Optimising convolutional neural networks using swarm intelligence,” in _UK Workshop on Computational Intelligence_. Springer, 2019, pp. 119–130.
  * [149] S. Lankford and D. Grimes, “Neural architecture search using particle swarm and ant colony optimization.” in _AICS_ , 2020, pp. 229–240.
  * [150] E. Suda and H. Iba, “Neural architecture search and weight adjustment by means of ant colony optimization,” in _2020 5th International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS)_. IEEE, 2020, pp. 184–191.
  * [151] R. Luo, F. Tian, T. Qin, E. Chen, and T.-Y. Liu, “Neural architecture optimization,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [152] M. Zhang, H. Li, S. Pan, X. Chang, and S. Su, “Overcoming multi-model forgetting in one-shot nas with diversity maximization,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 7806–7815.
  * [153] K. Kandasamy, W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing, “Neural architecture search with bayesian optimisation and optimal transport,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [154] H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, “Efficient neural architecture search via parameters sharing,” in _International conference on machine learning_. PMLR, 2018, pp. 4095–4104.
  * [155] Z. Ding, Y. Chen, N. Li, D. Zhao, Z. Sun, and C. L. P. Chen, “Bnas: Efficient neural architecture search using broad scalable architecture,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–15, 2021.
  * [156] Z. Lu, I. Whalen, V. Boddeti, Y. Dhebar, K. Deb, E. Goodman, and W. Banzhaf, “Nsga-net: Neural architecture search using multi-objective genetic algorithm,” in _Proceedings of the Genetic and Evolutionary Computation Conference_ , ser. GECCO ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 419–427. [Online]. Available: https://doi.org/10.1145/3321707.3321729
  * [157] Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y. Wei, and J. Sun, “Single path one-shot neural architecture search with uniform sampling,” in _European Conference on Computer Vision_. Springer, 2020, pp. 544–560.
  * [158] Z. Yang, Y. Wang, X. Chen, B. Shi, C. Xu, C. Xu, Q. Tian, and C. Xu, “Cars: Continuous evolution for efficient neural architecture search,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1829–1838.
  * [159] S. Mo, J. Xia, and P. Ren, “Simulated annealing for neural architecture search,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [160] M. Shahawy and E. Benkhelifa, “Hivenas: Neural architecture search using artificial bee colony optimization,” _arXiv preprint arXiv:2211.10250_ , 2022\.
  * [161] S. Hu, S. Xie, H. Zheng, C. Liu, J. Shi, X. Liu, and D. Lin, “Dsnas: Direct neural architecture search without parameter retraining,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 084–12 092.
  * [162] G. Kyriakides and K. Margaritis, “An introduction to neural architecture search for convolutional networks,” 2020. [Online]. Available: https://arxiv.org/abs/2005.11074
  * [163] Z. Chen and B. Li, “Efficient evolution for neural architecture search,” in _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2020, pp. 1–7.
  * [164] A. Klein, E. Christiansen, K. Murphy, and F. Hutter, “Towards reproducible neural architecture and hyperparameter search,” 2018. [Online]. Available: https://openreview.net/pdf?id=rJeMCSnml7
  * [165] P. Liu, M. D. El Basha, Y. Li, Y. Xiao, P. C. Sanelli, and R. Fang, “Deep evolutionary networks with expedited genetic algorithms for medical image denoising,” _Medical image analysis_ , vol. 54, pp. 306–315, 2019.
  * [166] Y. Xu, Y. Wang, K. Han, Y. Tang, S. Jui, C. Xu, and C. Xu, “Renas: Relativistic evaluation of neural architecture search,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4411–4420.
  * [167] A. Zela, A. Klein, S. Falkner, and F. Hutter, “Towards automated deep learning: Efficient joint neural architecture and hyperparameter search,” 2018\. [Online]. Available: https://arxiv.org/abs/1807.06906
  * [168] I. Trofimov, N. Klyuchnikov, M. Salnikov, A. Filippov, and E. Burnaev, “Multi-fidelity neural architecture search with knowledge distillation,” _arXiv preprint arXiv:2006.08341_ , 2020.
  * [169] S. Yang, Y. Tian, X. Xiang, S. Peng, and X. Zhang, “Accelerating evolutionary neural architecture search via multi-fidelity evaluation,” 2021. [Online]. Available: https://arxiv.org/abs/2108.04541
  * [170] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, “Hyperband: A novel bandit-based approach to hyperparameter optimization,” _The Journal of Machine Learning Research_ , vol. 18, no. 1, pp. 6765–6816, 2017.
  * [171] T. Chen, I. Goodfellow, and J. Shlens, “Net2net: Accelerating learning via knowledge transfer,” in _ICLR_ , 2016.
  * [172] Y. Weng, T. Zhou, Y. Li, and X. Qiu, “Nas-unet: Neural architecture search for medical image segmentation,” _IEEE Access_ , vol. 7, pp. 44 247–44 257, 2019.
  * [173] B. Baker, O. Gupta, R. Raskar, and N. Naik, “Accelerating neural architecture search using performance prediction,” in _ICLR_ , 2018.
  * [174] C. White, A. Zela, R. Ru, Y. Liu, and F. Hutter, “How powerful are performance predictors in neural architecture search?” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [175] R. Ru, C. Lyle, L. Schut, M. Fil, M. van der Wilk, and Y. Gal, “Speedy performance estimation for neural architecture search,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [176] T. Domhan, J. T. Springenberg, and F. Hutter, “Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves,” in _Twenty-fourth international joint conference on artificial intelligence_ , 2015.
  * [177] A. Klein, S. Falkner, J. T. Springenberg, and F. Hutter, “Learning curve prediction with bayesian neural networks,” in _ICLR 2017 (Poster)_ , 2017\.
  * [178] G. Bender, P.-J. Kindermans, B. Zoph, V. Vasudevan, and Q. Le, “Understanding and simplifying one-shot architecture search,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10–15 Jul 2018, pp. 550–559.
  * [179] R. K. Thomas, “Evolution of intelligence: An approach to its assessment,” _Brain, Behavior and Evolution_ , vol. 17, no. 6, pp. 454–472, 1980.
  * [180] E. Stern, “Individual differences in the learning potential of human beings,” _npj Science of Learning_ , vol. 2, no. 1, pp. 1–7, 2017.
  * [181] R. Boyd, P. J. Richerson, and J. Henrich, “The cultural niche: Why social learning is essential for human adaptation,” _Proceedings of the National Academy of Sciences_ , vol. 108, no. Supplement 2, pp. 10 918–10 925, 2011.
  * [182] E. Baum, M. Hutter, and E. Kitzelmann, “Artificial general intelligence,” in _Proceedings of the Third Conference on Artificial General Intelligence_. Citeseer, 2010.
  * [183] B. Goertzel, “Artificial general intelligence: concept, state of the art, and future prospects,” _Journal of Artificial General Intelligence_ , vol. 5, no. 1, p. 1, 2014.
  * [184] B. Widrow and M. A. Lehr, “30 years of adaptive neural networks: perceptron, madaline, and backpropagation,” _Proceedings of the IEEE_ , vol. 78, no. 9, pp. 1415–1442, 1990.
  * [185] R. Palnitkar and J. Cannady, “A review of adaptive neural networks,” in _IEEE SoutheastCon, 2004. Proceedings._ , 2004, pp. 38–47.
  * [186] M. Ghifary, W. B. Kleijn, and M. Zhang, “Domain adaptive neural networks for object recognition,” in _Pacific Rim international conference on artificial intelligence_. Springer, 2014, pp. 898–904.
  * [187] H. Zhang, S. Starke, T. Komura, and J. Saito, “Mode-adaptive neural networks for quadruped motion control,” _ACM Transactions on Graphics (TOG)_ , vol. 37, no. 4, pp. 1–11, 2018.
  * [188] K. O. Stanley, _Efficient evolution of neural networks through complexification_. The University of Texas at Austin, 2004.
  * [189] R. Ashmore, R. Calinescu, and C. Paterson, “Assuring the machine learning lifecycle: Desiderata, methods, and challenges,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 5, pp. 1–39, 2021.
  * [190] S. Munzert, C. Rubba, P. Meißner, and D. Nyhuis, _Automated data collection with R: A practical guide to web scraping and text mining_. John Wiley & Sons, 2014.
  * [191] Y. N. Kunang, S. D. Purnamasari _et al._ , “Web scraping techniques to collect weather data in south sumatera,” in _2018 International Conference on Electrical Engineering and Computer Science (ICECOS)_. IEEE, 2018, pp. 385–390.
  * [192] R. Diouf, E. N. Sarr, O. Sall, B. Birregah, M. Bousso, and S. N. Mbaye, “Web scraping: state-of-the-art and areas of application,” in _2019 IEEE International Conference on Big Data (Big Data)_. IEEE, 2019, pp. 6040–6042.
  * [193] M. Dojchinovski and T. Vitvar, “Linked web apis dataset,” _Semantic Web_ , vol. 9, no. 4, pp. 381–391, 2018.
  * [194] V. Krotov and L. Silva, “Legality and ethics of web scraping,” 2018.
  * [195] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [196] C. Bowles, L. Chen, R. Guerrero, P. Bentley, R. Gunn, A. Hammers, D. A. Dickie, M. V. Hernández, J. Wardlaw, and D. Rueckert, “Gan augmentation: Augmenting training data using generative adversarial networks,” _arXiv preprint arXiv:1810.10863_ , 2018.
  * [197] B. Wu, A. Wan, X. Yue, and K. Keutzer, “Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1887–1893.
  * [198] A. Kar, A. Prakash, M.-Y. Liu, E. Cameracci, J. Yuan, M. Rusiniak, D. Acuna, A. Torralba, and S. Fidler, “Meta-sim: Learning to generate synthetic datasets,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 4551–4560.
  * [199] P. Li, X. Rao, J. Blase, Y. Zhang, X. Chu, and C. Zhang, “Cleanml: a study for evaluating the impact of data cleaning on ml classification tasks,” in _2021 IEEE 37th International Conference on Data Engineering (ICDE)_. IEEE, 2021, pp. 13–24.
  * [200] S. Krishnan, M. J. Franklin, K. Goldberg, and E. Wu, “Boostclean: Automated error detection and repair for machine learning,” _arXiv preprint arXiv:1711.01299_ , 2017.
  * [201] S. Krishnan and E. Wu, “Alphaclean: Automatic generation of data cleaning pipelines,” _arXiv preprint arXiv:1904.11827_ , 2019.
  * [202] M. Volkovs, F. Chiang, J. Szlichta, and R. J. Miller, “Continuous data cleaning,” _2014 IEEE 30th International Conference on Data Engineering_ , pp. 244–255, 2014.
  * [203] C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” _Journal of big data_ , vol. 6, no. 1, pp. 1–48, 2019.
  * [204] X. Li, Y. Zhou, T. Wu, R. Socher, and C. Xiong, “Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 3925–3934.