Let us look at these studies more closely. We focus on the problem of bounding the number of examples needed to produce good generalizations when the learner faces a stream of tasks. Consider first that the goal of traditional learning is to find a hypothesis h ∗ ∈ H that minimizes a functional risk, h ∗ = arg minh∈H Rφ(h), where

$$R\_{\phi}(h) = \int\_{\left(\mathbf{x}, y\right) \in \mathcal{X} \times \mathcal{Y}} L(h(\mathbf{x}), y) d\phi(\mathbf{x}, y), \tag{12.7}$$

The risk corresponds to the expected loss incurred by hypothesis h; L(h(x), y) is a particular loss function (e.g., zero–one loss), and the integral runs across the input–output space. We introduce a new notation, φ, to denote the probability distribution over X × Y that indicates which examples are more likely to be seen for that particular task. Since we do not have access to all possible examples in the input–output space, we may choose to approximate the true risk with an empirical risk (Rˆφ(h)). We do this by randomly sampling m examples according to φ to generate a training sample T = {(xj , yj )} m j=1, where

$$\hat{R}\_{\phi}(h, T) = \frac{1}{m} \sum\_{j=1}^{m} L(h(\mathbf{x}\_{j}), y\_{j}). \tag{12.8}$$

It has been formally shown that one can bound the true risk Rφ(h) as a function of the empirical risk Rˆφ(h, T) if there exists a uniform bound for all h ∈ H on the probability of deviation between Rφ(h) and Rˆφ(h, T) (Vapnik, 1995; Blumer et al., 1989). Such bounds can be represented as a function of the Vapnik–Chervonenkis dimension of the hypothesis space H, VC(H). The VC dimension captures the degree of expressiveness or richness in delimiting flexible decision boundaries by the set of functions in H; it provides an objective characterization of H (Vapnik, 1995). Bounds for the deviation between Rφ(h) and Rˆφ(h, T) take on the form

$$R\_{\phi}(h) \le \hat{R}\_{\phi}(h, T) + g(m, \delta, \text{VC}(\mathcal{H})),\tag{12.9}$$

where the function g(·) explicitly indicates an upper bound on the distance between the true risk and the empirical risk; the inequality is satisfied for all h ∈ H with probability 1 − δ.

#### **12.4.1 The learning-to-learn scenario**

Let us now consider the novelty brought about by the learning-to-learn scenario (Baxter, 2000)). Here we assume the learner is embedded in a set of related tasks that share certain commonalities. In traditional learning, we assume a probability distribution φ that indicates which examples are more likely to be seen in such a task. Now we assume there is a metadistribution Φ over the space of all possible distributions {φ}. In essence, Φ indicates which tasks are more likely to be found within the sequence of tasks faced by the metalearner (just as φ indicates which examples are more likely to be seen in such a task). As an example, if we were interested in classifying luminous objects in astronomical surveys, Φ may stand for a probability distribution that peaks over tasks that identify classes of astronomical objects. Given a family of hypothesis spaces H, the goal of the metalearner is to find a hypothesis space H∗ ∈ H that minimizes a new functional risk, H∗ = arg minH∈H RΦ(H), where

$$R\_{\Phi}(\mathcal{H}) = \int\_{\phi \in \Phi} \inf\_{h \in \mathcal{H}} R\_{\phi}(h) d\Phi(\phi). \tag{12.10}$$

An expansion of the above formula gives

$$R\_{\Phi}(\mathcal{H}) = \int\_{\phi \in \Phi} \inf\_{h \in \mathcal{H}} \int\_{(\mathbf{x}, y) \in \mathcal{X} \times \mathcal{Y}} L(h(\mathbf{x}), y) d\phi(\mathbf{x}, y) d\Phi(\phi). \tag{12.11}$$

The new functional risk, RΦ(H), represents the expected loss of the best possible hypothesis in each hypothesis space. The integral runs across all task distributions {φ}, which are themselves distributed according to a metadistribution Φ. In practice, since we do not know the form of Φ, we need to draw samples T1, T2, . . . Tn to infer how tasks are distributed in our environment.

The advantage of working in a learning-to-learn scenario is that the learner accumulates experience after each new task. Such experience, here referred to as metaknowledge, is expected to result in more accurate models when the tasks share commonalities or patterns. The expectation is that, as more tasks are observed, the number of examples required to attain accurate models (with high probability) decreases over time.

#### **12.4.2 Bounds on generalization error for metalearners**

Finding bounds on the generalization error for metalearners follows the same logic as the one adopted in conventional learning theory. The idea is to formally show that it is possible to bound the new functional risk RΦ(H) as a function of the empirical risk RˆΦ(H). Given a set of n samples T = {Tp}, the empirical risk is defined as the average of the best possible empirical error for each training sample Tp:

$$\hat{R}\Phi(\mathcal{H}) = \frac{1}{n} \sum\_{p=1}^{n} \inf\_{h \in \mathcal{H}} \hat{R}\_{\phi}(h, T\_p). \tag{12.12}$$

The bound can be found if there exists a uniform bound for all H ∈ H on the probability of deviation between RΦ(H) and RˆΦ(H). In conventional learning theory, these bounds are governed by the expressiveness of the family of hypotheses H. Similarly, in the learning-to-learn scenario, bounds on the generalization error are governed by the size of the function classes associated with the family space H. Specifically, one can guarantee that with probability 1 − δ (according to the choice of samples T), all H ∈ H will satisfy the following inequality:

230 12 Transfer of Knowledge Across Tasks

$$R\_{\varPhi}(\varvarPi) \le \hat{R}\_{\varPhi}(\varvarPi) + \epsilon. \tag{12.13}$$

This holds if the number of tasks n is such that

$$n \ge \max\left\{\frac{256}{\epsilon^2} \log \frac{8\mathcal{C}(\frac{\epsilon}{32}, A\_{\mathbb{H}})}{\delta}, \frac{64}{\epsilon^2}\right\} \tag{12.14}$$

and the number of examples m for each task is such that

$$m \ge \max\left\{\frac{256}{n\epsilon^2} \log \frac{8\mathcal{C}(\frac{\epsilon}{32}, \mathbf{A}\_{\mathbb{H}}^n)}{\delta}, \frac{64}{\epsilon^2}\right\}.\tag{12.15}$$

The theorem (Baxter, 2000) introduces two new properties characterizing the family of hypothesis spaces H, C(, ΛH) and C(, Λ n H). These functions measure the capacity of H in a way similar to how the VC dimension measures the capacity of H. To provide continuity to our chapter, we defer explanation of these properties to Appendix A. The bounds stated above simply show that, to learn both a good hypothesis space H ∈ H and a good hypothesis h ∈ H, one needs a minimum number of both the number of tasks and the number of examples on each task. It is known that, if and δ are fixed (Baxter, 2000), the number of examples m needed on each task to attain an accurate model is such that

$$m = O\left(\frac{1}{n}\log \mathcal{C}(\epsilon, \mathbf{A}\_{\mathbb{H}}^{\mathbf{n}})\right). \tag{12.16}$$

This indicates that the required number of examples on each task decreases as the number of tasks increases, in accordance with our expectations of the benefits gained when the learning algorithm has the capability of exploiting previous experience.

#### **12.4.3 Other theoretical work**

#### **Bounds using algorithmic stability**

The results described above can be improved if one makes certain assumptions (Maurer, 2005). To understand this, we need to review the concept of algorithmic stability (Bousquet and Elisseeff, 2002). A learning algorithm is said to be uniformly β-stable if taking away one example from the training set does not modify the loss of the output hypothesis by more than β (for a fixed loss function). We update our definition of a metalearning algorithm as a function A(T) that outputs a hypothesis after looking at a sequence of samples T = {Tp} n p=1. That is, we no longer talk about a hypothesis space but of a single hypothesis that does well on all previous tasks. In that case, one can also think of a metalearning algorithm as being β 0 -stable if removing one sample from the set of samples T does not modify the loss of the output hypothesis by more than β 0 . Notice that the parameter β 0 corresponds to the concept of stability across tasks, whereas the parameter β is used to refer to stability across examples drawn from one task.

Given that A(T) = h for a given set of samples T, the new results show that, for every environment Φ, with probability greater than 1 − δ according to the selection of T, the following inequality holds:

$$\forall \phi \; R\phi \; (h) \le \frac{1}{n} \sum\_{p=1}^{n} \hat{R}\_{\phi\_p}(h, T\_p) + 2\beta' + (4n\beta' + m)\sqrt{\frac{\ln(1/\delta)}{2n}} + 2\beta,\tag{12.17}$$

where φp ∈ Φ and Rˆφp (h, Tp) is an estimation of the empirical loss of hypothesis h when the examples are drawn from the sample Tp. The first term on the right-hand side of the inequality is then the average empirical loss of h on the set of tasks T. It can be shown that the new bound is tighter than that of Section 12.4.2 (of course, under the assumption of stability parameterized by β and β 0 on A(T) = h).

#### **Bounds for domain adaptation**

The context of domain adaptation leads to another set of interesting learning bounds (Ben-David et al., 2010). Assume a source domain DS where class labels abound, and a target domain DT with few or no class labels. It is implicitly assumed that the source and target domains must be *related*, with no mechanism to quantify the degree of *relatedness*. This can be helpful to understand how to bound the error of a model trained on the source domain but applied to the target domain, where we assume the distribution over X has changed, i.e., PS(X) 6= PT (X).

We begin by defining the error of a hypothesis h under a zero–one loss function as Rφ(h) = E(x,y)∼φ[|h(x) − y|], where we assume Y = {−1, 1}. We refer to the source and target distributions as φS and φT , with the understanding that the only difference is in the marginal distributions PS(X) 6= PT (X). It has been formally shown that the generalization error on the target domain can be bound as a function of three terms:

$$R\_{\phi\_T}(h) \le R\_{\phi\_S}(h) + \frac{1}{2}d\mu\_{\ell\Delta\forall\ell}(\phi\_S, \phi\_T) + \lambda,\tag{12.18}$$

where the first term on the right hand side of the inequality simply refers to the generalization error on the source domain. The second term is a measure of the *relatedness* of the two distributions. Formally,

$$d\_{\mathcal{H}\Delta\mathcal{H}}(\phi\_S, \phi\_T) = 2 \sup\_{h, h' \in \mathcal{H}} |P\_{\mathbf{x} \sim \phi\_S}[h(\mathbf{x}) \neq h'(\mathbf{x})] - P\_{\mathbf{x} \sim \phi\_T}[h(\mathbf{x} \neq h'(\mathbf{x}))].\tag{12.19}$$

The goal is simply to capture the difference in the probability of disagreement between two hypotheses in the space of hypothesis H. The last term λ refers to the combined error of an *ideal* hypothesis:

$$
\lambda = R\_{\phi\_S}(h^\*) + R\_{\phi\_T}(h^\*), \tag{12.20}
$$

where h ∗ = argminh∈HRφS (h) + RφT (h). The bound depends, then, on the *distance* between the source and target distributions, and on the existence of a hypothesis that can attain low generalization error on both the source and target domains.

#### **12.4.4 Bias versus variance in metalearning**

As part of our theoretical study, we conclude by looking into the nature of the bias–variance dilemma in classification when immersed in a learning-to-learn framework. Let us first recall what the bias–variance dilemma states in traditional learning (Hastie et al., 2009; Geman et al., 1992). The dilemma is based on the fact that the expected prediction error, or risk, can be decomposed into bias and variance components.1 Ideally we would like to have classifiers with both low bias and low variance but these components are inversely related. On the one hand, simple classifiers encompass a small hypothesis space H. Their small repertoire of functions produces high bias (since the hypothesis with lowest prediction error may lie far from the true target function) but low variance (since there are

1A third component, the *irreducible error* or *Bayes error*, cannot be eliminated or traded (Hastie et al., 2009).

few hypotheses to choose from). On the other hand, increasing the size of H reduces the bias, but increases the variance. The large size of H normally allows for flexible decision boundaries (low bias), but the learning algorithm inevitably becomes sensitive to small variations in the data (high variance).

In the learning-to-learn framework, there is an equal need to find a balance in the size of the family of hypothesis spaces H. A small H will exhibit low variance and high bias; here, unless we can find a good hypothesis space H ∈ H with a small risk RΦ(H), the best H may be far from the true hypothesis space. And just as in traditional learning, a large H will exhibit low bias but high variance, since a large number of available hypothesis spaces increases the chances of selecting one that simply accommodates to the idiosyncrasies of the training data. One main goal is to understand if learning the right family of hypothesis spaces H is inherently easier (or not) than learning the right hypothesis space H.

## **Appendix A**

Section 12.4.2 makes use of two properties characterizing the space of a family of hypothesis spaces H, C(, ΛH) and C(, Λ n H). These functions quantify the capacity of the space of a family of hypothesis spaces H. We now explain the nature of these properties in more detail:2

**Definition 1**. For each H ∈ H, define a new function λH(φi) by

$$\lambda\_{\mathcal{H}}(\phi) = \inf\_{h \in \mathcal{H}} R\_{\phi}(h), \tag{12.21}$$

where λ : Φ → [0, 1]. In other words, the function λ specifies the minimum error loss achieved after looking at every h ∈ H under the distribution φ.

**Definition 2**. For the family of hypothesis spaces H, define a new set ΛH by

$$A\_{\mathbb{H}} = \{\lambda \mathbb{A} : \mathcal{H} \in \mathbb{H}\}.\tag{12.22}$$

According to Definition 1, the set ΛH contains all *different* functions within the space of a family of hypotheses H. We can compute the expected difference in the minimum error loss for any two functions λ1, λ2 ∈ ΛH as follows:

**Definition 3**. For any two functions λ1, λ2 ∈ ΛH and a distribution Φ on the space of possible input–output distributions, define

$$D\_{\Phi}(\lambda\_1, \lambda\_2) = \int\_{\phi} |\lambda\_1(\phi) - \lambda\_2(\phi)| d\Phi(\phi). \tag{12.23}$$

The function D can be seen as the expected distance between two functions λ1, λ2. We now define the concept of an -cover as follows:

**Definition 4**. An -cover of (ΛH, DΦ) is a set {λ1, λ2, · · · , λn} such that, for all λ ∈ ΛH, DΦ(λ, λp) ≤ (1 ≤ p ≤ n). Let N (, ΛH, DΦ) represent the size of the smallest -cover. We now define the capacity of ΛH by

2We follow Baxter's work (Baxter, 2000) in different order and notation to simplify the explanation of the two properties characterizing H.

$$\mathcal{L}(\epsilon, A\_{\mathbb{H}}) = \sup\_{\Phi} \mathcal{N}(\epsilon, A\_{\mathbb{H}}, D\_{\Phi}), \tag{12.24}$$

where the supremum runs over all probability distributions over X × Y.

We can similarly define the second capacity C(, Λ n H). To begin, consider a sequence of n tasks that has been modeled with n hypotheses h = (h1, h2, · · · , hn). We can compute the expected error loss across n tasks as follows:

$$\lambda\_{\mathbf{h}}^{n}(\{\mathbf{x}, y\}) = \frac{1}{n} \sum\_{p=1}^{n} L(h\_{p}(\mathbf{x}), y). \tag{12.25}$$

**Definition 5**. For the space of a family of hypotheses H, define a new set Λ n h by

$$A\_{\mathbf{h}}^{n} = \{\lambda\_{\mathbf{h}}^{n} : h\_1, h\_2, \dots, h\_n \in \mathcal{H}\}.\tag{12.26}$$

The set Λ n h is a loss function class and, as before, it indicates how many *different* classes of functions (capturing the average error loss for a sequence of n hypotheses) are contained within the hypothesis space H; the difference is that now we are comparing sets of n loss functions.

**Definition 6**. For the space of a family of hypotheses H, define

$$\mathbf{A}\_{\mathbb{H}}^{\mathbf{n}} = \bigcup\_{\mathcal{H} \in \mathbb{H}} A\_{\mathbf{h}}^{\mathbf{n}},\tag{12.27}$$

where h ⊆ H. The second capacity C(, Λ n H) is defined similarly to the first one but using a new distance function:

$$D^n\_{\Phi}(\mathbf{h}, \mathbf{h}') = \int\_{(\mathcal{X} \times \mathcal{Y})^n} |\lambda^n\_{\mathbf{h}}(\{\mathbf{x}\_i, y\_i\}) - \lambda^n\_{\mathbf{h}'}(\{\mathbf{x}\_i, y\_i\})| d\phi\_1, d\phi\_2, \cdots, d\phi\_n. \tag{12.28}$$

This brings us to the second capacity function:

$$\mathcal{C}(\epsilon, \mathbf{A}\_{\mathbb{H}}^{\mathbf{n}}) = \sup\_{\phi} \mathcal{N}(\epsilon, \mathbf{A}\_{\mathbb{H}}^{\mathbf{n}}, D\_{\Phi}^{\mathbf{n}}),\tag{12.29}$$

where the supremum runs over all sequences of n probability distributions over X × Y.

## **References**

Torrey, L. and Shavlik, J. (2010). Transfer learning. In *Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques*, pages 242– 264. IGI Global.

Vapnik, V. (1995). *The Nature of Statistical Learning Theory*. Springer Verlag, New York.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Metalearning for Deep Neural Networks**

Mike Huisman, Jan N. van Rijn, and Aske Plaat

**Summary.** Deep neural networks have enabled large breakthroughs in various domains ranging from image and speech recognition to automated medical diagnosis. However, these networks are notorious for requiring large amounts of data to learn from, limiting their applicability in domains where data is scarce. Through metalearning, the networks can learn how to learn, allowing them to learn from fewer data. In this chapter, we provide a detailed overview of metalearning for knowledge transfer in deep neural networks. We categorize the techniques into (i) metric-based, (ii) model-based, and (iii) optimization-based techniques, cover the key techniques per category, discuss open challenges, and provide directions for future research such as performance evaluation on heterogeneous benchmarks.

## **13.1 Introduction**

Although current deep learning methods obtain great successes, training is generally time consuming and large datasets are required to achieve good performance. Metalearning offers a solution to these issues. That is, metalearning deep neural networks can improve their learning ability over time, or equivalently, "learn to learn": this allows them to learn new concepts from less data.

Most metalearning techniques, in the context of deep learning, learn at two levels. At the *inner level*, the agent is presented with a new task (dataset) and tries to quickly learn the associated concepts. This quick adaptation is facilitated by the knowledge that the agent has accumulated from other tasks, at the *outer level*. Thus, the inner level involves a single task, whereas the outer level involves a multitude of tasks. The accumulated knowledge is often directly embedded into the parameters of the agent (neural network). Note that this is in contrast to some other methods in this book (see, e.g., Chapter 6), where metalearning is used to optimize the hyperparameters of algorithms.

In this chapter, we provide a detailed overview of metalearning for knowledge transfer in deep neural networks, which was briefly introduced in the previous chapter. Following Vinyals (2017), we categorize this field into three groups: (i) metric-, (ii) model-, and (iii) optimization-based techniques. After introducing our notation and providing background information, we summarize key techniques of each category, identify the main challenges, and formulate open questions.

## **13.2 Background and Notation**

In this section, we introduce and contrast base-level learning and metalearning in the context of deep learning. Additionally, we briefly discuss common training and evaluation procedures.

#### **13.2.1 The meta-abstraction for deep neural networks**

In *supervised learning*, we wish to learn a function fθ : X → Y that maps inputs xi ∈ X to their corresponding outputs yi ∈ Y . In the context of deep learning, f is a neural network with parameters θ. Note that this is in contrast to previous chapters, where θ was used to denote hyperparameters of algorithms. Given a dataset D = {(xi, yi)} m i=1 of m examples, learning boils down to finding parameters θ that minimize an empirical loss function LD. This loss function captures how well the model is performing by computing the difference between the model predictions yˆi and correct dataset outputs yi. In short, we wish to find the optimal parameters

$$\boldsymbol{\theta}^\* := \arg\min\_{\boldsymbol{\theta}} \mathcal{L}\_D(\boldsymbol{\theta}).\tag{13.1}$$

Finding theoretically optimal parameters θ ∗ is often infeasible. We can, however, approximate them, guided by *pre-defined* meta-knowledge ω (including, e.g., initial model parameters θ, choice of optimizer, and learning rate schedule) (Hospedales et al., 2020). As such, we approximate

$$\boldsymbol{\theta}^\* \approx g\_{\omega}(\boldsymbol{D}, \mathcal{L}\_D),\tag{13.2}$$

where gω is an optimization procedure that takes meta-knowledge ω, a dataset D, and loss function LD to produce updated weights gω(D, LD) that presumably perform well on D. In practice, the optimizer gω often uses gradients of the loss function to update the model parameters θ. To measure the generalization performance of the found parameters θ ∗ , we can split the dataset into training and test sets D tr and D test, and use, e.g., cross-validation techniques (see Chapter 3).

In contrast, *supervised metalearning* for deep neural networks does not assume that some meta-knowledge ω is given, or pre-defined. Instead, the goal is to find the best ω such that new *tasks* Tj (datasets) can be learned as quickly as possible. Metalearning techniques often learn ω from a multitude of tasks Tj . Note that this in contrast to regular supervised learning, where only one task (dataset) is used.

More formally, we have a probability distribution of tasks p(T ) and wish to find optimal meta-knowledge

$$\omega^\* := \operatorname\*{arg\,min}\_{\omega} \underbrace{\mathbb{E}\_{\mathcal{T}\_j \sim p(\mathcal{T})} [\mathcal{L}\_{\mathcal{T}\_j} (g\_\omega(\mathcal{T}\_j, \mathcal{L}\_{\mathcal{T}\_j}))]}\_{\text{tner level}}.\tag{13.3}$$

Here, the inner level concerns task-specific learning, while the outer level concerns multiple tasks. One can now easily see why this is metalearning: we learn ω, which allows for quick learning of tasks Tj at the inner level. Hence, we are learning to learn.

Fig. 13.1: Illustration of N-way k-shot classification, where N = 5, and k = 1. Meta-validation tasks are not displayed. Adapted from Ravi and Larochelle (2017)

#### **13.2.2 Common training and evaluation procedures**

In the previous subsection, we described the learning objectives for supervised learning and metalearning for deep neural networks. However, we remained agnostic with respect to the setup used in practice to achieve these objectives. In general, one optimizes a meta-objective by using various tasks. In practice, this is done in three stages: the (i) *meta-train*, (ii) *meta-validation*, and (iii) *meta-test* stages, each of which is associated with a set of tasks from a homogeneous source. Importantly, note that this is in contrast to some of the previous chapters, where heterogeneous data sources were used for knowledge transfer.

Now, in the meta-train stage, the metalearning algorithm is deployed on the metatrain tasks. The meta-validation tasks can then be used to evaluate the performance on unseen tasks which were not used for training. Effectively, this measures the *metageneralization* ability of the trained network, which serves as feedback to tune, e.g., hyperparameters of the metalearning algorithm. Lastly, the meta-test tasks are used to give a final performance estimate of the metalearning technique.

#### N**-way** k**-shot learning**

A frequently used instantiation of this general meta-setup is called N-way k-shot classification (see Figure 13.1). This setup is also divided into the three stages — meta-train, meta-validation, and meta-test — which are used for metalearning, metalearner hyperparameter optimization, and evaluation, respectively. Each stage has a corresponding set of disjoint labels, i.e., L tr, Lval, Ltest ⊂ Y , such that L tr ∩L val = ∅, Ltr ∩L test = ∅, and L val ∩ L test = ∅. In a given stage s, *tasks/episodes* Tj = (D tr Tj , Dtest Tj ) are obtained by sampling examples (xi, yi) from the full dataset D, such that every yi ∈ L s . Note that this requires access to a dataset D. Now, the sampling process is guided by the N-way kshot principle, which states that every train dataset D tr Tj should contain exactly N classes and k examples per class, implying that |D tr Tj | = N · k. Furthermore, the true labels of examples in the test set D test Tj must be present in the training set D tr Tj of a given task Tj . D tr T j acts as a *support set*, literally supporting classification decisions on the *query set* D test Tj . Throughout this chapter, we will use the terms training/support and test/query sets interchangeably. Importantly, note that with this terminology, the test set (or query set) of a task is actually used during the meta-training phase. Furthermore, the fact that the labels across stages are disjoint ensures that we test the ability of a model to learn *new* concepts.

The metalearning objective in the training phase is to minimize the loss function of the model predictions on the query sets, conditioned on the support sets. As such, for a given task, the model "sees" the support set and extracts information from the support set to guide its predictions on the query set. By applying this procedure for different episodes/tasks, with different support and query sets, the model will slowly accumulate meta-knowledge ω, which can ultimately speed up learning on new tasks.

At the meta-validation and meta-test stages, or evaluation phases, the learned metainformation in ω is fixed. The model is, however, still allowed to make task-specific updates to its parameters θ (which implies that it is learning). After task-specific updates, we can evaluate the performance on the test sets. This way we test how well a technique performs at metalearning.

N-way k-shot classification is often performed for small values of k (since we want our models to learn new concepts quickly, i.e., from few examples). In that case, one can refer to it as *few-shot learning*.

#### **13.2.3 Overview of the rest of this chapter**

In the remainder of this chapter we will look in more detail at individual metalearning methods. As indicated before, the methods can be grouped into three main categories (Vinyals, 2017), which we will discuss in sequence:

To help give an overview of the methods, we draw your attention to the following tables. Table 13.1 summarizes the three categories and provides the key ideas, strengths, and weaknesses of the approaches. The technical details are explained in the remainder of this chapter. Table 13.2 contains an overview of all the techniques that are discussed further on.

## **13.3 Metric-Based Metalearning**

At a high level, the goal of metric-based techniques is to acquire — among others meta-knowledge ω in the form of a good feature space that can be used for various new tasks. In the context of neural networks, this feature space coincides with the weights θ of the networks. Then, new tasks can be learned by comparing new inputs with example inputs (of which we know the labels) in the metalearned feature space. The greater the similarity between a new input and an example, the more likely it is that the new input will have the same label as the example input.

Metric-based techniques are a form of metalearning as they leverage their prior learning experience (metalearned feature space) to "learn" new tasks more quickly. Here,

Table 13.1: High-level overview of the categorization of metalearning techniques for deep neural networks. See Section 13.6 for more details. Table extended from Vinyals (2017)

Table 13.2: Overview of the metalearning techniques discussed in this chapter

#### 242 13 Metalearning for Deep Neural Networks

"learn" is used in a non-standard way since metric-based techniques do not make any network changes when presented with new tasks, as they rely solely on input comparisons in the already metalearned feature space. These input comparisons are a form of *non-parametric learning*; i.e., new task information is not absorbed into the network parameters.

More formally, metric-based learning techniques aim to learn a similarity kernel, or equivalently, *attention mechanism* kθ (parameterized by θ), that takes two inputs x1 and x2 and outputs their similarity score. Larger scores indicate greater similarity. Class predictions for new inputs x can then be made by comparing x with example inputs xi, of which we know the true label yi, the underlying idea being that the greater the similarity between x and xi, the more likely it becomes that x also has label yi.

Given a task Tj = (D tr Tj , Dtest Tj ) and an unseen input vector x ∈ D test Tj , a probability distribution over classes Y is computed/predicted as a weighted combination of labels from the support set D tr Tj , using the similarity kernel kθ:

$$P\theta(Y|x, D\_{\mathcal{T}\_j}^{tr}) = \sum\_{(x\_i, y\_i) \in D\_{\mathcal{T}\_j}^{tr}} k\theta(x, x\_i) y\_i. \tag{13.4}$$

Importantly, the labels yi are assumed to be *one-hot encoded*, meaning that they are represented by zero vectors with a "1" at the position of the true class. For example, suppose there are five classes in total, and our example x1 has true class 4. Then, the one-hot encoded label is y1 = [0, 0, 0, 1, 0]. Note that the probability distribution Pθ(Y |x, Dtr Tj ) over classes is a vector of size |Y |, in which the ith entry corresponds to the probability that input x has class Yi (given the support set). The predicted class is thus yˆ = argmaxi=1,2,...,|Y |Pθ(Y |x, S)i, where Pθ(Y |x, S)i is the computed probability that input x has class Yi.

Fig. 13.2: Illustration of our metric-based example

#### **Example**

Suppose that we are given a task Tj = (D tr Tj , Dtest Tj ). Furthermore, suppose that D tr Tj = {([0, −4], 1),([−2, −4], 2),([−2, 4], 3),([6, 0], 4)}, where a tuple denotes a pair (xi, yi). For simplicity, the example will not use an embedding function, which maps example inputs onto an (more informative) embedding space. Now, our test set only contains one example D test Tj = {([4, 0.5], y)}. Then, the goal is to predict the correct label for the new input [4, 0.5] using only examples in D tr Tj . The problem is visualized in Figure 13.2, where red vectors correspond to example inputs from our training set. The blue vector is the new input that needs to be classified. Intuitively, this new input is most similar to the vector [6, 0], which means that we expect the label for the new input to be the same as that for [6, 0], i.e., 4.

Now, suppose we use a fixed similarity kernel, namely the cosine similarity, i.e., k(x, xi) = x·x T i ||x||·||xi|| , where ||v|| denotes the length of vector v, i.e., ||v|| = p ( P n v 2 n). Here, vn denotes the nth element of placeholder vector v (substitute v by x or xi). We can now compute the cosine similarity between the new input [4, 0.5] and every example input xi, as done in Table 13.3, where we used the facts that ||x|| = || [4, 0.5] || = √ 4 2 + 0.5 2 ≈ 4.03, and x ||x|| ≈ [4,0.5] 4.03 = [0.99, 0.12].

From this table and Equation 13.4, it follows that the predicted probability distribution Pθ(Y |x, Dtr Tj ) = −0.12y1 − 0.58y2 − 0.37y3 + 0.99y4 = −0.12[1, 0, 0, 0] − 0.58[0, 1, 0, 0] − 0.37[0, 0, 1, 0] + 0.99[0, 0, 0, 1] = [−0.12, −0.58, −0.37, 0.99]. Note that this is not really a probability distribution. That would require normalization such that every element is at least 0 and the sum of all elements is 1. For the sake of this example, we do not perform this normalization, as it is clear that class 4 (the class of the most similar example input [6, 0]) will be predicted.

Table 13.3: Example showing pairwise input comparisons. Numbers were rounded to two decimals

One may wonder why such techniques are metalearners, for we could take any single dataset D and use pairwise comparisons to compute predictions. Now, at the outer level, metric-based metalearners are trained on a distribution of different tasks, in order to learn (among others) a good input embedding function. This embedding function facilitates inner level learning, which is achieved through pairwise comparisons. As such, one learns an embedding function across tasks to facilitate task-specific learning, which is equivalent to "learning to learn", or metalearning.

In the remainder of this section we will discuss various key metric-based techniques, including

#### **13.3.1 Siamese neural networks**

A Siamese neural network (Koch et al., 2015) consists of two neural networks fθ that share the same weights θ. Siamese neural networks take two inputs x1, x2, and compute two hidden states fθ(x1), fθ(x2), corresponding to the activation patterns in the final hidden layers. These hidden states are fed into a distance layer, which computes a distance vector d = |fθ(x1) − fθ(x2)|, where di is the absolute distance between the ith elements of fθ(x1) and fθ(x2). From this distance vector, the similarity between x1, x2 is computed as σ(α T d), where σ is the sigmoid function (with output range [0,1]), and α is a vector of free weighting parameters, determining the importance of each di. This network structure can be seen in Figure 13.3.

Fig. 13.3: Example of a Siamese neural network. Source: Koch et al. (2015)

Koch et al. (2015) applied this technique to few-shot image recognition in two stages. In the first stage, they train the twin network on an *image verification* task, where the goal is to output whether two input images x1 and x2 have the same class. The network is thus stimulated to learn discriminative features. In the second stage, where the model is confronted with a new task, the network leverages its prior learning experience. That is, given a task Tj = (D tr Tj , Dtest Tj ) and previously unseen input x ∈ D test Tj , the predicted class yˆ is equal to the label yi of the example (xi, yi) ∈ D tr Tj which yields the highest similarity score to x. In contrast to other techniques mentioned further in this chapter, Siamese neural networks do not optimize directly for good performance across tasks. However, they do leverage learned knowledge from the verification task to learn new tasks quicker.

In summary, Siamese neural networks are a simple and elegant approach to perform few-shot learning. However, they are not readily applicable outside the supervised learning setting.

#### **13.3.2 Matching networks**

Matching networks (Vinyals et al., 2016) build upon the idea that underlies Siamese neural networks (Koch et al., 2015). That is, they leverage pairwise comparisons between the given support set D tr Tj = {(xi, yi)} m i=1 (for a task Tj ) and new inputs x ∈ D test Tj from the query/test set which we want to classify. However, instead of assigning the class yi of the most similar example input xi, matching networks use a weighted combination of *all* the example labels yi in the support set, based on the similarity of inputs xi to new input x. More specifically, predictions are computed as follows: yˆ = Pm i=1 a(x, xi)yi, where a is a non-parametric (non-trainable) attention mechanism, or similarity kernel. This classification process is shown in Figure 13.4. In this figure, the input to fθ has to be classified, using the support set D tr Tj (input to gθ).

Fig. 13.4: Architecture of matching networks. Source: Vinyals et al. (2016)

The attention that is used consists of a softmax over the cosine similarity c between the inputs, i.e.,

$$a(x, x\_i) = \frac{e^{c(f\_{\Phi}(x), g\_{\Phi}(x\_i))}}{\sum\_{j=1}^{m} e^{c(f\_{\Phi}(x), g\_{\Phi}(x\_j))}},\tag{13.5}$$

where fφ and gϕ are neural networks, parameterized by φ and ϕ, that map raw inputs to a (lower-dimensional) latent vector (corresponding to the activation state in the final hidden layer of a neural network). As such, neural networks act as embedding functions. Now, the larger the cosine similarity between the embeddings of x and xi, the larger a(x, xi) and thus the influence of label yi on the predicted label yˆ for input x.

Vinyals et al. (2016) propose two main choices for the embedding functions. The first is to use a single neural network, granting us θ = φ = ϕ and thus fφ = gϕ. This setup is the default form of matching networks, as shown in Figure 13.4. The second choice is to make fφ and gϕ dependent on the support set D tr Tj using long short-term memory networks (LSTMs). In that case, fφ is represented by an attention LSTM, and gϕ by a bidirectional one. This choice for embedding functions is called *full context embeddings* (FCE), and yielded an accuracy improvement of roughly 2% on miniImageNet compared with the regular matching networks, indicating that task-specific embeddings can aid the classification of new data points from the same distribution.

Matching networks learn a good feature space across tasks for making pairwise comparisons between inputs. In contrast to Siamese neural networks (Koch et al., 2015), this feature space is learned across tasks instead of on a distinct verification task.

In summary, matching networks are an elegant and simple approach to metric-based metalearning. However, these networks are not readily applicable outside of supervised learning settings, and suffer in performance when label distributions are biased (Vinyals et al., 2016).

Some slight variations on matching networks have given rise to new metric-based techniques. Prototypical networks (Snell et al., 2017) use Euclidean distance as the basis for a similarity function, and reduce the required number of pairwise comparisons by comparing the new input x with class prototypes (mean vectors of inputs xi from a class) in the support set. Instead of using fixed similarity metrics, relation networks (Sung et al., 2018) learn a similarity metric represented by a neural network, which allows for greater expressive power.

#### **13.3.3 Graph neural networks**

Graph neural networks (Garcia and Bruna, 2017) use a more general and flexible approach than previously discussed techniques for N-way k-shot classification. As such, graph neural networks subsume Siamese (Koch et al., 2015) and prototypical networks (Snell et al., 2017). The graph neural network approach represents each task as a fully connected graph G = (V, E), where V is a set of nodes/vertices and E a set of edges connecting nodes. In this graph, nodes vi correspond to input embeddings fθ(xi), concatenated with their one-hot encoded labels yi, i.e., vi = [fθ(xi), yi]. For inputs x from the query/test set (for which we do not have the labels), a uniform prior over all N possible labels is used: y = [ 1 N , . . . , 1 N ]. Thus, each node contains an input and label section. Edges are weighted links that connect these nodes.

The graph neural network then propagates information in the graph using a number of local operators. The underlying idea is that label information can be transmitted from nodes for which we do have the labels to nodes for which we have to predict labels. The used local operators are out of scope for this chapter, and the reader is referred to Garcia and Bruna (2017) for details.

By exposing the graph neural network to various tasks Tj , the propagation mechanism can be altered to improve the flow of label information in such a way that predictions become more accurate. As such, in addition to learning a good input representation function fθ, graph neural networks also learn to propagate label information from labeled examples to unlabeled inputs.

Graph neural networks achieve good performance in few-shot settings (Garcia and Bruna, 2017) and are not restricted to supervised learning settings.

Fig. 13.5: Processing in an attentive recurrent comparator. Source: Shyam et al. (2017)

#### **13.3.4 Attentive recurrent comparators**

Attentive recurrent comparators (Shyam et al., 2017) differ from previously discussed techniques as they do not compare inputs as a whole, but by parts. This approach is inspired by how humans would make a decision concerning the similarity of objects. That is, we shift our attention from one object to the other, and move back and forth to take glimpses of different parts of both objects. In this way information of two objects is fused from the beginning, whereas other techniques (e.g., matching networks (Vinyals et al., 2016) and graph neural networks (Garcia and Bruna, 2017)) only combine information at the end (after embedding both images) (Shyam et al., 2017).

Given two inputs xi and x, we feed them in interleaved fashion repeatedly into a recurrent neural network (controller): xi, x, . . . , xi, x. Thus, the image at time step t is given by It = xi if t is even, else x. Then, at each time step t, the attention mechanism focuses on a square region of the current image: Gt = attend(It, Ωt), where Ωt = Wght−1 are attention parameters, which are computed from the previous hidden state ht−1. The next hidden state ht+1 = RNN(Gt, ht−1) is given by the glimpse at time t, i.e., Gt, and the previous hidden state ht−1. The entire sequence consists of g glimpses per image. After this sequence is fed into the recurrent neural network (indicated by RNN(. . .)), the final hidden state h2g is used as a combined representation of xi relative to x. This process is summarized in Figure 13.5. Classification decisions can then be made by feeding the combined representations into a classifier. Optionally, the combined representations can be processed by bi-directional LSTMs before passing them to the classifier.

The attention approach is biologically inspired and biologically plausible. A downside of attentive recurrent comparators is the higher computational cost, while the performance is often not better than less biologically plausible techniques such as graph neural networks (Garcia and Bruna, 2017).

#### **Metric-based techniques, in conclusion**

In this section, we have seen various metric-based techniques. The metric-based techniques metalearn an informative feature space that can be used to compute class predictions based on input similarity scores.

Key advantages of these techniques are that (i) the underlying idea of similaritybased predictions is conceptually simple, and (ii) they can be fast at test time when tasks are small, as the networks do not need to make task-specific adjustments. However, when tasks at meta-test time become more distant from the tasks that were used at metatraining time, metric-learning techniques are unable to absorb new task information into the network weights. Consequently, performance may suffer.

Furthermore, when tasks become larger, pairwise comparisons may become computationally expensive. Lastly, most metric-based techniques rely on the presence of labeled examples, which makes them inapplicable outside supervised learning settings.

## **13.4 Model-Based Metalearning**

A different approach to metalearning for deep neural networks is the model-based approach. On a high level, model-based techniques rely upon an adaptive, internal state, in contrast to metric-based techniques, which generally use a fixed neural network at test time.

More specifically, model-based techniques maintain a stateful, internal representation of a task. When presented with a task, a model-based neural network processes the support/training set in sequential fashion. At every time step, an input enters and alters the internal state of the model. Thus, the internal state can capture relevant task-specific information, which can be used to make predictions for new inputs.

Because the predictions are based on internal dynamics that are hidden from the outside, model-based techniques are also called *black boxes*. Information from previous inputs must be remembered, which is why model-based techniques have a memory component, either in- or externally.

Recall that the mechanics of metric-based techniques were limited to pairwise input comparisons. This is not the case for model-based techniques, where the human designer has the freedom to choose the internal dynamics of the algorithm. As a result, modelbased techniques are not restricted to metalearning good feature spaces, as they can also learn internal dynamics, used to process and predict the input data of tasks.

More formally, given a support set D tr Tj corresponding to task Tj , model-based techniques compute a class probability distribution for a new input x as follows:

$$P\theta(Y|x, D\_{\mathcal{T}\_j}^{tr}) = f\_{\theta}(x, D\_{\mathcal{T}\_j}^{tr}),\tag{13.6}$$

where f represents the black-box neural network model and θ its parameters.

#### **Example**

Using the same example as in Section 13.3, suppose we are given a task training set D tr Tj = {([0, −4], 1),([−2, −4], 2),([−2, 4], 3),([6, 0], 4)}, where a tuple denotes a pair (xi, yi). Furthermore, suppose our test set only contains one example D test Tj = {([4, 0.5], 4)}. This problem has been visualized in Figure 13.2 (in Section 13.3). Now, for the sake of the example, we do not use an input embedding function: our model will operate on the raw inputs of D tr Tj and D test Tj . As an internal state, our model uses an external *memory matrix* M ∈ R 4×(2+1), with four rows (one for each example in our support set) and three columns (the dimensionality of input vectors, plus one dimension for the correct label). Our model proceeds to process the support set in sequential fashion, reading the examples from D tr Tj one by one, and by storing the ith example in the ith row of the memory module. After processing the support set, the memory matrix contains all examples and, as such, serves as an internal task representation.

Now, given the new input [4, 0.5], our model could use many different techniques to make a prediction based on this representation. For simplicity, assume that it computes the dot product between x and every memory M(i) (the 2-D vector in the ith row of M, ignoring the correct label) and predicts the class of the input which yields the largest dot product. This would produce scores of −2, −10, −6, and 24 for the examples in D tr Tj , respectively. Since the last example [6, 0] yields the largest dot product, we predict that class, i.e., 4.

This example was a bit simplistic for illustrative purposes. More advanced and successful techniques have been proposed, including

We discuss these techniques in order.

#### **13.4.1 Memory-augmented neural networks**

The key idea of memory-augmented neural networks (Santoro et al., 2016) is to enable neural networks to learn quickly with the help of an *external memory*. The main *controller* (the recurrent neural network interacting with the memory) then gradually accumulates knowledge across tasks, while the external memory allows for quick taskspecific adaptation. For this, Santoro et al. (2016) used neural Turing machines (Graves et al., 2014). Here, the controller is parameterized by θ and acts as the long-term memory of the memory-augmented neural network, while the external memory module is the short-term memory.

The workflow of memory-augmented neural networks is displayed in Figure 13.6. Note that the data from a task is processed as a sequence; i.e., data are fed into the network one by one. The support/training set is fed into the memory-augmented neural network first. Afterwards, the query/test set is processed. At time step t, the model receives input xt with the label of the previous input, i.e., yt−1. This was done to prevent the network from mapping class labels directly to the output (Santoro et al., 2016).

The interaction between the controller and memory is visualized in Figure 13.7. Given an input xt at time t, the controller generates a key kt, which is used to either retrieve or store a memory from/in memory matrix M, i.e., Mt(i). To read from memory using a key kt, a (column) read vector w r t is created with the same number of rows as the memory matrix M, where each entry i denotes the cosine similarity between the key kt, and the memory stored in row i, i.e., Mt(i). Then, the memory rt = P i w r t (i)M(i) is retrieved, which is simply a linear combination of rows/memories in the memory matrix M.

Fig. 13.6: Workflow of memory-augmented neural networks. Source: Santoro et al. (2016)

Fig. 13.7: Controller–memory interaction in memory-augmented neural networks. Source: Santoro et al. (2016)

To write to memory, Santoro et al. (2016) propose a new mechanism called least recently used access (LRUA). LRUA writes to either the least, or most recently used memory location. In the former case, it preserves recent memories, and in the latter it updates recently obtained information. The writing mechanism works by keeping track of how often every memory location is accessed in a usage vector w u t , which is updated at every time step according to the following update rule: w u t := γw u t−1 + w r t + w w t , where the superscripts u, w, and r refer to usage, write, and read vectors, respectively. In words, the previous usage vector is decayed (using parameter γ), while current reads (w r t ) and writes (w w t ) are added to the usage. Now, let n be the total number of reads to memory, and `u(n) (`u for 'least used') be the nth smallest value in the usage vector w u t . Then, the least-used weights are defined as follows:

$$w\_t^{\ell u}(i) = \begin{cases} 0 & \text{if } w\_t^u(i) > \ell u(n) \\ 1 & \text{else} \end{cases}.$$

Then, the write vector w w t is computed as w w t = σ(α)w r t−1 + (1 − σ(α))w `u t−1, where α is a parameter that interpolates between the two weight vectors. As such, if σ(α) = 1, we write to the most recently used memory, whereas when σ(α) = 0, we write to the least recently used memory locations. Finally, writing is performed as follows: Mt(i) := Mt−1(i) + w w t (i)kt, for all i.

Predictions are made as follows: given an input xt, memory-augmented neural networks compute the corresponding memory rt and feed that memory into a classifier. Across tasks, memory-augmented neural networks learn a good input embedding function fθ and classifier weights, which can be exploited when presented with new tasks.

In summary, memory-augmented neural networks (Santoro et al., 2016) combine an external memory and a neural network to achieve metalearning. The interaction between a controller, with long-term memory parameters θ, and the memory M may also be interesting for studying human metalearning (Santoro et al., 2016). In contrast to many metric-based techniques, this model-based technique is applicable to both classification and regression problems. A downside of this approach is the architectural complexity.

#### **13.4.2 Meta networks**

Fig. 13.8: Architecture of a meta network. Source: Munkhdalai and Yu (2017)

In a similar fashion to memory-augmented neural networks (Santoro et al., 2016), meta networks (Munkhdalai and Yu, 2017) also leverage the idea of an external memory module. However, meta networks use the memory for a different purpose. Meta networks are divided into two distinct subsystems (consisting of neural networks), i.e., the base- and metalearner (whereas in memory-augmented neural networks the baseand meta-components are intertwined). Now, the base-learner is responsible for performing tasks, and for providing the metalearner with meta-information, such as loss gradients. The metalearner can then compute fast task-specific weights for itself and the base-learner, such that it can perform better on the given task Tj = (D tr Tj , Dtest Tj ). This workflow is depicted in Figure 13.8.

The metalearner consists of neural networks uφ, mϕ, and dψ. The network uφ is used as an input representation function. The networks dψ and mϕ are used to compute task-specific weights φ ∗ and example-level fast weights θ ∗ . Lastly, bθ is the base-learner which performs input predictions. Note that we used the term "fast weights" throughout, which refers to task- or input-specific versions of slow (initial) weights.

1: Sample S = {(xi, yi) v D tr Tj } T i=1 from the support set 2: **for** (xi, yi) ∈ S **do** 3: Li = error(uφ(xi), yi) 4: **end for** 5: φ ∗ = dψ({∇φLi} T i=1) 6: **for** (xi, yi) ∈ D tr Tj **do** 7: Li = error(bθ(xi), yi) 8: θ ∗ i = mϕ(∇θLi) 9: Store θ ∗ i in ith position of example-level weight memory M 10: ri = uφ,φ∗ (xi) 11: Store ri in ith position of representation memory R 12: **end for** 13: Ltask = 0 14: **for** (x, y) ∈ D test Tj **do** 15: r = uφ,φ∗ (x) 16: a = attention(R, r) {ak is the cosine similarity between r and R(k)} 17: θ ∗ = softmax(a) TM 18: Ltask = Ltask + error(bθ,θ∗ (x), y) 19: **end for** 20: Update Θ = {θ, φ, ψ, ϕ} using ∇ΘLtask **Algorithm 13.1:** Meta networks, by Munkhdalai and Yu (2017)

The pseudocode for meta networks is displayed in Algorithm 13.1. First, a sample of the support set is created (line 1), which is then used to compute task-specific weights φ ∗ for the representation network u (lines 2–5).

Then, meta networks iterate over every example (xi, yi) in the support set D tr Tj . The base-learner bθ attempts to make class predictions for these examples, resulting in loss values Li (lines 7–8). The gradients of these losses are used to compute fast weights θ ∗ for example i (line 8), which are then stored in the ith row of memory matrix M (line 9). Additionally, input representations ri are computed and stored in the memory matrix R (lines 10–11).

Now, meta networks are ready to address the query set D test Tj . They iterate over every example (x, y) and compute a representation r of it (line 15). This representation is matched against the representations of the support set, which are stored in the memory matrix R. This matching gives us a similarity vector a, where every entry k denotes the similarity between the input representation r and the kth row in the memory matrix R, i.e., R(k) (line 16). A softmax over this similarity vector is performed to normalize the entries. The resulting vector is used to compute a linear combination of weights that were generated for inputs in the support set (line 17). These weights θ ∗ are specific for the input x in the query set, and can be used by the base-learner b to make predictions for that input (line 18). The observed error is added to the task loss. After the entire query set is processed, all the involved parameters can be updated using backpropagation (line 20).

Fig. 13.9: Layer augmentation setup used to combine slow and fast weights. Source: Munkhdalai and Yu (2017)

Note that some neural networks use both slow and fast weights at the same time. Munkhdalai and Yu (2017) use an augmentation setup for this, as depicted in Figure 13.9.

In short, meta networks rely on a reparameterization of the meta- and base-learner for every task. Despite the flexibility and applicability to both supervised and reinforcement learning settings, the approach is quite complex. It consists of many components, each with its own set of parameters, which can be a burden on memory usage and computation time. Additionally, finding the correct architecture for all the involved components can be time consuming.

#### **13.4.3 Simple neural attentive meta-learner (SNAIL)**

Instead of an external memory matrix, SNAIL (Mishra et al., 2018) relies on a special model architecture to serve as the memory. Mishra et al. (2018) argue that it is not possible to use recurrent neural networks for this, as they have limited memory capacity and cannot pinpoint specific prior experiences (Mishra et al., 2018). Hence, SNAIL uses a different architecture, consisting of 1D *temporal convolutions* (Oord et al., 2016) and a *soft attention* mechanism (Vaswani et al., 2017). The temporal convolutions allow for 'high-bandwidth" memory access, and the attention mechanism allows us to pinpoint specific experiences. Figure 13.10 visualizes the architecture and workflow of SNAIL for supervised learning problems. From this figure, it becomes clear why this technique is model based. That is, model outputs are based upon the internal state, computed from earlier inputs.

Fig. 13.10: Architecture and workflow of SNAIL. Temporal convolution blocks are orange; attention blocks are green. Source: Mishra et al. (2018)

SNAIL consists of three building blocks. The first is the *DenseBlock*, which applies a single 1D convolution to the input and concatenates (in the feature/horizontal direction) the result. The second is a *TCBlock*, which is simply a series of *DenseBlocks* with exponentially increasing dilation rate of the temporal convolutions (Mishra et al., 2018). Note that the dilation is nothing but the temporal distance between two nodes in a network. For example, if we use a dilation of 2, a node at position p in layer L will receive the activation from node p − 2 from layer L − 1. The third block is the *AttentionBlock*, which learns to focus on the important parts of prior experience.

In a similar fashion to memory-augmented neural networks (Santoro et al., 2016) (Subsection 13.4.1), SNAIL also processes task data in sequence, as shown in Figure 13.10. However, the input at time t is accompanied with the label at time t, instead of t−1 (as was the case for memory-augmented neural networks). SNAIL learns internal dynamics from seeing various tasks, so that it can make good predictions on the query set, conditioned upon the support set.

A key advantage of SNAIL is that it can be applied to both supervised and reinforcement learning tasks. In addition, it achieves good performance compared with previously discussed techniques. A downside of SNAIL is that finding the correct architecture of TCBlocks and DenseBlocks can be time consuming.

#### **13.4.4 Conditional neural processes**

Fig. 13.11: Conditional neural process. Source: Garnelo et al. (2018)

In contrast to previous techniques, a conditional neural process (CNP) (Garnelo et al., 2018) does not rely on an external memory module. Instead, it aggregates the support set into a single aggregated latent representation. The general architecture is shown in Figure 13.11. As we can see, the conditional neural process operates in three phases on task Tj . First, it observes the training set D tr Tj , including the ground-truth outputs yi. Examples (xi, yi) ∈ D tr Tj are embedded using a neural network hθ into representations ri. Second, these representations are aggregated using the operator a to produce a single representation r of D tr Tj (hence it is model based). Third, a neural network gφ processes this single representation r and the new inputs x and produces the predictions yˆ.

Let the entire conditional neural process model be denoted by QΘ, where Θ is a set of all the involved parameters {θ, φ}. The training process is different compared with other techniques. Let xTj and yTj denote all inputs and corresponding outputs in D tr Tj . Then, the first ` v U(0, . . . , k·N −1) examples in D tr Tj are used as a conditioning set D c Tj (effectively splitting the training set in a true training set and a validation set). Given a value of `, the goal is to maximize the log likelihood (or minimize the negative log likelihood) of the labels yTj in the entire training set D tr Tj :

$$\mathcal{L}(\boldsymbol{\Theta}) = -\mathbb{E}\_{\mathcal{T}\_j \sim p(\boldsymbol{\mathcal{T}})} \left[ \mathbb{E}\_{\ell \sim U(0, \dots, k \cdot N - 1)} \left( Q\_{\boldsymbol{\Theta}}(\boldsymbol{y}\_{\mathcal{T}\_j} | \boldsymbol{D}^c\_{\boldsymbol{\mathcal{T}}\_j}, \boldsymbol{x}\_{\mathcal{T}\_j}) \right) \right]. \tag{13.7}$$

Conditional neural processes are trained by repeatedly sampling various tasks and values of `, and propagating the observed loss backwards.

In summary, conditional neural processes use compact representations of previously seen inputs to aid the classification of new observations. Despite its simplicity and elegance, a disadvantage of this technique is that it is often outperformed in few-shot settings by other techniques such as matching networks (Vinyals et al., 2016) (see Subsection 13.3.2).

#### **Model-based techniques, in conclusion**

In this section, we have discussed various model-based techniques. Despite apparent differences, they all build on the notion of task internalization; that is, tasks are processed and represented in the state of the model-based system. This state can then be used to make predictions.

Advantages of model-based approaches include the flexibility of the internal dynamics of the systems and their broader applicability compared with most metricbased techniques. However, model-based techniques are often outperformed by metricbased techniques in supervised settings (e.g., graph neural networks (Garcia and Bruna, 2017); Subsection 13.3.3), may not perform well when presented with larger datasets (Hospedales et al., 2020), and generalize less well to more distant tasks than optimizationbased techniques (Finn and Levine, 2018). We discuss this optimization-based approach next.

## **13.5 Optimization-Based Metalearning**

Optimization-based techniques adopt a different perspective on metalearning than the previous two approaches. They explicitly optimize for fast learning. Most optimizationbased techniques do so by approaching metalearning as a bilevel optimization problem. At the inner level, a base-learner makes task-specific updates using some optimization strategy (such as gradient descent). At the outer level, the performance across tasks is optimized.

More formally, given a task Tj = (D tr Tj , Dtest Tj ) with new a input x ∈ D test Tj and base-learner parameters θ, optimization-based metalearners return

$$P(Y|x, D\_{\mathcal{T}\_j}^{tr}) = f\_{\theta\_{\varphi(\theta, D\_{\mathcal{T}\_j}^{tr}, \mathcal{L}\_{\mathcal{T}\_j})}}(x),\tag{13.8}$$

where f is the base-learner and gϕ is a (learned) optimizer that makes task-specific updates to the base-learner parameters θ using the training data D tr Ti and loss function LTj .

#### **Example**

Suppose we are faced with a linear regression problem, where every task is associated with a different function f(x). For this example, suppose our model only has two parameters: a and b, which together form the function ˆf(x) = ax+b. Suppose further that our meta-training set consists of four different tasks, i.e., A, B, C, and D. Then, according to the optimization-based view, we wish to find a single set of parameters {a, b} from which we can quickly learn the optimal parameters for each of the four tasks, as displayed in Figure 13.12. In fact, this is the intuition behind the popular optimization-based technique MAML (Finn et al., 2017).

We now discuss the following optimization-based techniques, in turn:

#### **13.5.1 LSTM optimizer**

Standard gradient update rules have the form

Fig. 13.12: Example of an optimization-based technique, inspired by Finn et al. (2017)

$$\theta\_{t+1} := \theta\_t - \alpha \nabla \theta\_t \mathcal{L} \tau\_j(\theta\_t), \tag{13.9}$$

where α is the learning rate and LTj (θt) is the loss function with respect to the task Tj and network parameters at time t, i.e., θt. The key idea underlying LSTM optimizers (Andrychowicz et al., 2016) is to replace the update term (−α∇LTj (θt)) by an update proposed by an LSTM g with parameters ϕ. Then, the new update becomes

$$\theta\_{t+1} := \theta\_t + g\_{\varphi}(\nabla\_{\theta\_t} \mathcal{L}\_{\overline{\gamma}\_j}(\theta\_t)). \tag{13.10}$$

Fig. 13.13: Workflow of the LSTM optimizer. Gradients can only propagate backwards through solid edges. ft denotes the observed loss at time step t. Source: Andrychowicz et al. (2016)

This new update allows the optimization strategy to be tailored to a specific family of tasks. Note that this is metalearning; i.e., the LSTM learns to learn.

#### 258 13 Metalearning for Deep Neural Networks

The loss function used to train an LSTM optimizer is

$$\mathcal{L}(\varphi) = \mathbb{E}\_{\mathcal{L}\_{\mathcal{T}\_j}} \left[ \sum\_{t=1}^{T} w\_t \mathcal{L}\_{\mathcal{T}\_j}(\theta\_t) \right],\tag{13.11}$$

where T is the number of parameter updates made, and wt are weights larger than zero (set to the constant 1 in the original paper (Andrychowicz et al., 2016)). As is often done, second-order derivatives (arising from the dependency between the updated weights and the LSTM optimizer) were ignored due to the computational expense associated with the computation thereof. This loss function is fully differentiable and thus allows for training an LSTM optimizer (see Figure 13.13). To prevent a parameter explosion, the same network is used for every *coordinate*/weight in the base-learner's network, causing the update rule to be the same for every parameter. Of course, the updates depend on their prior values and gradients.

The key advantage of LSTM optimizers is that they can enable faster learning compared with hand-crafted optimizers, also on different datasets than those used to train the optimizer. However, Andrychowicz et al. (2016) did not apply this technique to fewshot learning. In fact, they did not apply it across tasks at all. Thus, it is unclear whether this technique can perform well in few-shot settings, where few data per class are available for training. Furthermore, the question remains of whether it can scale to larger base-learner architectures.

#### **13.5.2 Reinforcement learning optimizer**

Li and Malik (2018) proposed a framework which casts optimization as a reinforcement learning problem. Optimization can then be performed by existing reinforcement learning techniques. Now, at a high level, an optimization algorithm g takes as input an initial set of weights θ0 and a task Tj with a corresponding loss function LTj and produces a sequence of new weights θ1, . . . , θT , where θT is the final solution found. On this sequence of proposed new weights, we can define a loss function L that captures unwanted properties (e.g., slow convergence, oscillations, etc.). The goal of learning an optimizer can then be formulated more precisely as follows: we wish to learn an optimal optimizer g ∗ :

$$g^\* = \operatorname{argmin}\_{g} \mathbb{E}\_{\mathcal{T}\_j \sim p(\mathcal{T}), \mathbf{e}\_0 \sim p(\theta\_0)} [\mathcal{L}(g(\mathcal{L}\_{\mathcal{T}\_j}, \theta\_0))].\tag{13.12}$$

The key insight is that the optimization can be formulated as a partially observable Markov decision process (POMDP). Then, the state corresponds to the current set of weights θt, the action to the proposed update at time step t, i.e., ∆θt, and the policy to the function that computes the update. With this formulation, the optimizer g can be learned by existing reinforcement learning techniques. In their paper, they used a recurrent neural network as the optimizer. At each time step, they feed it observation features, which depend on the previous set of weights, loss gradients, and objective functions, and use guided policy search to train it.

In summary, Li and Malik (2018) made a first step towards general optimization through reinforcement learning optimizers, which were shown to be able to generalize across network architectures and datasets. However, the base-learner architecture that was used was quite small. The question remains of whether this approach can scale to larger architectures.

Fig. 13.14: MAML learns an initialization point from which it can perform well on various tasks. Source: Finn et al. (2017)

#### **13.5.3 Model-agnostic meta-learning (MAML)**

Model-agnostic meta-learning (MAML) (Finn et al., 2017) uses a simple gradient-based inner optimization procedure (e.g., stochastic gradient descent) instead of more complex LSTM procedures or procedures based on reinforcement learning. The key idea of MAML is to explicitly optimize for fast adaptation to new tasks by learning a good set of initialization parameters θ. This is shown in Figure 13.14: from the learned initialization θ, we can quickly move to the best set of parameters for task Tj , i.e., θ ∗ j for j = 1, 2, 3. The learned initialization can be seen as the *inductive bias* of the model, or simply the set of assumptions (encapsulated in θ) that the model makes with respect to the overall task structure.

More formally, let θ denote the initial model parameters of a model. The goal is to quickly learn new concepts, which is equivalent to achieving a minimal loss in few gradient update steps. The amount of gradient steps s has to be specified upfront, such that MAML can explicitly optimize for achieving good performance within that number of steps. Suppose we pick only one gradient update step, i.e., s = 1. Then, given a task Tj = (D tr Tj , Dtest Tj ), gradient descent would produce updated parameters (i.e., fast weights)

$$\boldsymbol{\theta}'\_{j} = \boldsymbol{\theta} - \alpha \nabla\_{\boldsymbol{\theta}} \mathcal{L}\_{D\frac{tr}{\boldsymbol{\tau}\_{j}}}(\boldsymbol{\theta}),\tag{13.13}$$

specific to task i. The *meta-loss* of quick adaptation (using s = 1 gradient steps) across tasks can then be formulated as

$$ML := \sum\_{\mathcal{T}\_j \sim p(\mathcal{T})} \mathcal{L}\_{D\_{\mathcal{T}\_j}^{test}}(\boldsymbol{\theta}\_j') = \sum\_{\mathcal{T}\_j \sim p(\mathcal{T})} \mathcal{L}\_{D\_{\mathcal{T}\_j}^{test}}(\boldsymbol{\theta} - \alpha \nabla \boldsymbol{\theta} \mathcal{L}\_{D\_{\mathcal{T}\_j}^{tx}}(\boldsymbol{\theta})),\tag{13.14}$$

where p(T ) is a probability distribution over tasks. This expression contains an inner gradient (∇θLTj (θj )). As such, by optimizing this meta-loss using gradient-based techniques, we have to compute second-order gradients. One can easily see this in the computation below:

#### 260 13 Metalearning for Deep Neural Networks

$$\begin{split} \nabla\_{\theta} ML &= \nabla\_{\theta} \sum\_{\mathcal{T}\_{j} \sim p(\mathcal{T})} \mathcal{L}\_{D\_{\mathcal{T}\_{j}}^{test}}(\theta\_{j}') \\ &= \sum\_{\mathcal{T}\_{j} \sim p(\mathcal{T})} \nabla\_{\theta} \mathcal{L}\_{D\_{\mathcal{T}\_{j}}^{test}}(\theta\_{j}') \\ &= \sum\_{\mathcal{T}\_{j} \sim p(\mathcal{T})} \mathcal{L}\_{D\_{\mathcal{T}\_{j}}^{test}}'(\theta\_{j}') \nabla\_{\theta}(\theta\_{j}') \\ &= \sum\_{\mathcal{T}\_{j} \sim p(\mathcal{T})} \mathcal{L}\_{D\_{\mathcal{T}\_{j}}^{test}}'(\theta\_{j}') \nabla\_{\theta}(\theta - \alpha \nabla \theta \mathcal{L}\_{D\_{\mathcal{T}\_{j}}^{tx}}(\theta)) \\ &= \underbrace{\sum\_{\mathcal{T}\_{j} \sim p(\mathcal{T})} \mathcal{L}\_{D\_{\mathcal{T}\_{j}}^{test}}'(\theta\_{j}')}\_{\text{FOMAL}}(\theta\_{j}') (\nabla\_{\theta}\theta - \alpha \nabla\_{\theta}^{2} \mathcal{L}\_{D\_{\mathcal{T}\_{j}}^{tx}}(\theta)), \tag{13.15} \end{split} \tag{13.15}$$

where we used L 0 Dtest Tj (θ 0 j ) to denote the derivative of the loss function with respect to the test set, evaluated at the post-update parameters θ 0 j . The term α∇2 θLDtr Tj (θ) contains the second-order gradients. The computation thereof is expensive in terms of time and memory costs, especially when the optimization trajectory is large (when using a larger number of gradient updates s per task). Finn et al. (2017) experimented with leaving out second-order gradients, by assuming ∇θθ 0 j = I, giving us first-order MAML (FOMAML, see Equation 13.15). They found that FOMAML performed reasonably similar implying that most of the benefit comes from the post-update gradients. This means that updating the initialization using only first-order gradients P Tjvp(T ) L 0 Dtest Tj (θ 0 j ) is roughly equal to using the full gradient expression for the meta-loss in Equation 13.15. One can extend the meta-loss to incorporate multiple gradient steps by substituting θ 0 j by a multi-step variant.

Now, MAML is trained as follows: the initialization weights θ are updated by continuously sampling a batch of m tasks B = {Tj v p(T )} m i=1. Then, for every task Tj ∈ B, an *inner update* is performed to obtain θ 0 j , in turn granting an observed loss LDtest Tj (θ 0 j ).

These losses across a batch of tasks are used in the *outer update*:

$$\boldsymbol{\theta} := \boldsymbol{\theta} - \beta \nabla\_{\boldsymbol{\theta}} \sum\_{\boldsymbol{\mathcal{T}}\_{j} \in B} \mathcal{L}\_{D\_{j}^{test}}(\boldsymbol{\theta}\_{j}^{\prime}). \tag{13.16}$$

The complete training procedure of MAML is displayed in Algorithm 13.2. At test time, when presented with a new task Tj , the model is initialized with θ and performs a number of gradient updates on the task data. Note that the algorithm for FOMAML is equivalent to Algorithm 13.2, except for the fact that the update on line 8 is done differently. That is, FOMAML updates the initialization with the rule θ = θ − β P Tjvp(T ) L 0 Dtest Tj (θ 0 j ).

Antoniou et al. (2019), in response to MAML, proposed many technical enhancements that can improve the training stability, performance, and generalization ability. Improvements include (i) updating the initialization θ after every inner update step (instead of after all steps are done) to increase gradient propagation, (ii) using secondorder gradients only after 50 epochs to increase the training speed, (iii) learning layerwise learning rates to improve flexibility, (iv) annealing the metalearning rate β over time, and (v) some batch normalization tweaks (keep running statistics instead of batchspecific ones, and using per-step biases).

9: **end while**

**Algorithm 13.2:** One-step MAML for supervised learning, by Finn et al. (2017)

MAML has attracted great attention within the field of metalearning for deep neural networks, perhaps due to its (i) simplicity (only requires two hyperparameters), (ii) general applicability, and (iii) strong performance. A downside of MAML, as mentioned above, is that it can be quite expensive in terms of running time and memory to optimize a base-learner for every task and compute higher-order derivatives from the optimization trajectories.

Several other optimization-based techniques build upon MAML. Here, we briefly mention some of them. *Meta-SGD* (Li et al., 2017) does not only learn an initialization point, but also appropriate learning rates for every single parameter in the network. *Latent embedding optimization* (LEO) (Rusu et al., 2018) attempts to find a good initialization in a lower-dimensional space, which can aid generalization performance. Several probabilistic versions of MAML have also been proposed, which learn a distribution over initializations (Grant et al., 2018; Finn et al., 2018) or multiple initializations which are jointly optimized (Yoon et al., 2018). Lastly, one can combine a representation module (e.g., a CNN) with closed-form base-learners, for which one can derive analytical solutions (Bertinetto et al., 2019; Lee et al., 2019).

#### **13.5.4 Reptile**

Reptile (Nichol and Schulman, 2018) is another optimization-based technique that, like MAML (Finn et al., 2017), solely attempts to find a good set of initialization parameters θ. The way in which Reptile attempts to find this initialization is quite different from MAML. It repeatedly samples a task, trains on the task, and moves the model weights towards the trained weights (Nichol and Schulman, 2018). Algorithm 13.3 displays the pseudocode describing this simple process.

```
6: end for
```
**Algorithm 13.3:** Reptile, by Nichol and Schulman (2018)

#### 262 13 Metalearning for Deep Neural Networks

Nichol and Schulman (2018) note that it is possible to treat (θ − θ 0 j )/α as gradients, where α is the learning rate of the inner stochastic gradient descent optimizer (line 4 in the pseudocode) and to feed that into a meta-optimizer (e.g., Adam). Moreover, instead of sampling one task at a time, one could sample a batch of n tasks and move the initialization θ towards the average update direction θ¯ = 1 n Pn j=1(θ 0 j − θ), granting the update rule θ := θ + θ¯.

The intuition behind Reptile is that updating the initialization weights towards updated parameters will grant a good inductive bias for tasks from the same family. By performing Taylor expansions of the gradients of Reptile and MAML (both first order and second order), Nichol and Schulman (2018) show that the expected gradients differ in their direction. They argue, however, that in practice, the gradients of Reptile will also bring the model towards a point minimizing the expected loss over tasks.

A mathematical argument as to why Reptile works goes as follows: let θ denote the initial parameters and θ ∗ j the optimal set of weights for task Tj . Lastly, let d be the Euclidean distance function. Then, the goal is to minimize the distance between the initialization point θ and the optimal point θ ∗ j :

$$\min\_{\theta} \mathbb{E}\_{\mathcal{T}\_j \sim p(\mathcal{T})} [\frac{1}{2} d(\theta, \theta\_j^\*)^2]. \tag{13.17}$$

Fig. 13.15: Visualization of Reptile's learning trajectory. Source: (Nichol and Schulman, 2018)

The gradient of this expected distance with respect to the initialization θ is given by

$$\begin{split} \nabla \boldsymbol{\theta} \mathbb{E}\_{\mathcal{T}\_{\mathcal{I}} \sim p(\mathcal{T})} [\frac{1}{2} d(\boldsymbol{\theta}, \boldsymbol{\theta}\_{j}^{\star})^{2}] &= \mathbb{E}\_{\mathcal{T}\_{\mathcal{I}} \sim p(\mathcal{T})} [\frac{1}{2} \nabla \boldsymbol{\theta} d(\boldsymbol{\theta}, \boldsymbol{\theta}\_{j}^{\star})^{2}] \\ &= \mathbb{E}\_{\mathcal{T}\_{\mathcal{I}} \sim p(\mathcal{T})} [\boldsymbol{\theta} - \boldsymbol{\theta}\_{j}^{\star}], \end{split} \tag{13.18}$$

where we used the fact that the gradient of the squared Euclidean distance between two points x1 and x2 is the vector 2(x1 − x2). Nichol and Schulman (2018) go on to argue that performing gradient descent on this objective would result in the following update rule:

$$\begin{split} \boldsymbol{\theta} &= \boldsymbol{\theta} - \epsilon \nabla\_{\theta} \frac{1}{2} d(\boldsymbol{\theta}, \boldsymbol{\theta}\_{j}^{\*})^{2} \\ &= \boldsymbol{\theta} - \epsilon (\boldsymbol{\theta}\_{j}^{\*} - \boldsymbol{\theta}). \end{split} \tag{13.19}$$

Since we do not know θ ∗ Tj , one can approximate this term by k steps of gradient descent SGD(LTj , θ, k). In short, Reptile can be seen as gradient descent on the distance minimization objective given in Equation 13.17. A visualization is shown in Figure 13.15. The initialization θ (φ in the figure) is moving towards the optimal weights for tasks 1 and 2 in interleaved fashion (hence the oscillations).

In conclusion, Reptile is an extremely simple metalearning technique, which does not need to differentiate through the optimization trajectory like, e.g., MAML (Finn et al., 2017), saving time and memory costs. However, the theoretical foundation is a bit weaker, and performance may be a bit worse than that of MAML.

#### **Optimization-based techniques, in conclusion**

Optimization-based techniques explicitly optimize for fast learning. A key advantage of optimization-based approaches is that they can achieve better performance on wider task distributions than, e.g., model-based approaches (Finn and Levine, 2018). However, optimization-based techniques optimize a base-learner for every task that they are presented with, which is computationally expensive (Hospedales et al., 2020).

## **13.6 Discussion and Outlook**

In this section, we give a helicopter view of the methods discussed in this chapter, and the field of metalearning for deep neural networks in general.

In recent years, this field has gained much popularity. The idea of self-improving neural networks that can leverage prior learning experience to learn new tasks more quickly is quite appealing. Instead of training a new model from scratch for different tasks, we can use the same (metalearning) model across tasks. As such, metalearning can widen the applicability of powerful deep learning techniques to domains where less data is available and computational resources are limited.

Metalearning techniques for deep neural networks are characterized by their metaobjective, which allows them to maximize performance across various tasks, instead of a single one, as is the case in base-level learning objectives. This meta-objective is reflected in the training procedure of metalearning methods, as they learn on a set of different meta-training tasks. The few-shot setting lends itself nicely towards this end, as tasks consist of few data points. This makes it computationally feasible to train on many different tasks, and it allows us to evaluate whether a neural network can learn new concepts from few examples. Task construction for training and evaluation does require some special attention to prevent the *memorization problem* (meta-overfitting), where the neural network has memorized tasks seen at training time but fails to generalize to new tasks. Clever task design and meta-regularization may prove useful to avoid such problems (Yin et al., 2020). Furthermore, to improve test performance, it is beneficial to match training and test conditions (Vinyals et al., 2016), and perhaps train in a more difficult setting than the one that will be used for evaluation (Snell et al., 2017).

On a high level, there are three categories of metalearning in the context of deep learning, namely (i) metric-, (ii) model-, and (iii) optimization-based ones, which rely on computing input similarity, task embeddings with states, and task-specific updates, respectively. Each approach has strengths and weaknesses. Metric-learning techniques are simple and effective (Garcia and Bruna, 2017) but are not readily applicable outside of the supervised learning setting (Hospedales et al., 2020). Model-based techniques, on the other hand, can have very flexible internal dynamics but lack generalization ability to more distant tasks than the ones used at meta-training time (Finn and Levine, 2018). Optimization-based approaches have shown greater generalizability but are in general computationally expensive, as they optimize a base-learner for every task (Finn and Levine, 2018; Hospedales et al., 2020).

Table 13.1 provides a concise, tabular overview of these approaches. Many techniques have been proposed for each one of the categories, and the underlying ideas may vary greatly, even within the same category. Table 13.2, therefore, provides an overview of all methods and key ideas that we have discussed in this chapter. Recall that Tj is a task, D tr Tj the corresponding training set, LTj the loss function, kθ(x, xi) a kernel function returning the similarity between the two inputs x and xi, yi are true labels for example inputs xi, θ are base-learner parameters, and gϕ is a (learned) optimizer with parameters ϕ.

#### **13.6.1 Open challenges**

Despite the great potential of metalearning for deep neural networks, there are still open challenges. In addition to the challenges mentioned above (computational costs and the memorization problem), there are two other major challenges. Firstly, most of the metalearning techniques discussed in this chapter are evaluated on narrow benchmark sets. This means that the data that the metalearner used for training are not too distant from the data used for evaluating its performance. As such, one may wonder how well these techniques are actually able to adapt to more distant tasks. Chen et al. (2019) showed that the ability to adapt to new tasks decreases as they become more distant from the tasks seen at training time. Increasing the degree of transferability of prior learning experience is an important and open challenge. As noted by Hospedales et al. (2020), the recently proposed meta-dataset (Triantafillou et al., 2019), could prove to be a great tool towards this end.

Secondly, Chen et al. (2019) have shown that a simple non-metalearning baseline yields competitive or better performance than some metalearning techniques on fewshot image classification. This begs the question of whether metalearning for deep neural networks is a good approach at all in the few-shot setting.

#### **13.6.2 Future research**

Future research is needed to address these challenges. Moreover, it is interesting to further investigate the applicability of metalearning for deep neural networks to online, active, and continual/lifelong learning settings (see, e.g., Finn et al. (2018); Yoon et al. (2018); Finn et al. (2019); Munkhdalai and Yu (2017); Vuorio et al. (2018)), as that could increase the applicability of deep learning tools in the real world.

Yet another direction for future research is the creation of *compositional* metalearning systems, which instead of learning flat and associative functions x → y, organize knowledge in a *compositional* manner. This would allow them to decompose an input x into several (already learned) components c1(x), . . . , cn(x), which in turn could help the performance in low-data regimes (Tokmakov et al., 2019). Lastly, the question has been raised of whether some contemporary metalearning techniques for deep neural networks actually learn how to perform rapid learning, or simply learn a set of robust high-level features, which can be (re)used for many (new) tasks. Raghu et al. (2020) investigated this question for MAML (Finn and Levine, 2018) and found that it largely relies on feature reuse.

We end this chapter by formulating a list of open research questions concerning the challenges that we encounter when applying metalearning to deep neural networks:

Metalearning for deep neural networks is a vibrant field. Despite great advances, ample challenges remain to be solved, which makes for exciting future work.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Automating Data Science**

**14**

**Summary.** It has been observed that, in data science, a great part of the effort usually goes into various preparatory steps that precede model-building. The aim of this chapter is to focus on some of these steps. A comprehensive description of a given task to be resolved is usually supplied by the domain expert. Techniques exist that can process natural language description to obtain task descriptors (e.g., keywords), determine the task type, the domain, and the goals. This in turn can be used to search for the required domain-specific knowledge appropriate for the given task. In some situations, the data required may not be available and a plan needs to be elaborated regarding how to get it. Although not much research has been done in this area so far, we expect that progress will be made in the future. In contrast to this, the area of preprocessing and transformation has been explored by various researchers. Methods exist for selection of instances and/or elimination of outliers, discretization and other kinds of transformations. This area is sometimes referred to as *data wrangling*. These transformations can be learned by exploiting existing machine learning techniques (e.g., learning by demonstration). The final part of this chapter discusses decisions regarding the appropriate level of detail (granularity) to be used in a given task. Although it is foreseeable that further progress could be made in this area, more work is needed to determine how to do this effectively.

## **14.1 Introduction**

It is well known in data science that a greater part of the effort goes into various preparatory operations, while the model-building step typically requires less effort. This has motivated researchers to examine how to automate the steps that precede the model building. In this chapter our aim is to analyze these preparatory steps. They include:

Some steps (e.g., step 5) were discussed in detail in some of the previous chapters. A brief overview is given in Section 14.6. Step 4 will be discussed further on.

As for steps 1, 2, and 3, these are normally carried out by data science specialists and many people believe that it is difficult to automate them. This may be the reason why these steps were not included among the topics of various recent workshops on the topic of automating data science.1 Although automation is difficult at this stage, it is nevertheless interesting to consider how this could be done and as a result provide assistance to data scientists.

Step 1, the issue of defining problems (or tasks), is discussed in Section 14.2. It is foreseeable that, as more experience is gathered from many different domains, some common patterns could be identified which in turn could provide a basis for at least partial automation. Section 14.3 discusses step 2: the problem of identifying useful domainspecific knowledge that could be included in the training data or the learning process itself. Section 14.4 is dedicated to step 3, discussing various strategies used for obtaining the training data. As pointed out in that section, it may happen that the data may not already be available in some application areas, and so a process must be devised regarding how to overcome this.

Section 14.5 discusses data preprocessing. This includes many types of transformations that are already well covered by AutoML methods, such as feature selection, but also *data wrangling*, which is not so easily automated but which has attracted a great deal of attention recently. Evidence of this are various workshops (e.g., AutoDS 2019 (Bie et al., 2019)) and research projects (e.g., AIDA (The Alan Turing Institute, 2020)). This section discusses also *data aggregation operations*, which are essential in many data mining applications.

Our motivation was to advance the overall understanding of this area and to share this with others. We hope that, this way, we can contribute to the development of future decision support systems in this area or even partial automation of the processes involved.

## **14.2 Defining the Current Problem/Task**

The formal definition of a given problem (task) involves a number of steps which can be summarized as follows:

Each step is discussed in detail in the respective subsection.

## **14.2.1 Problem understanding and description**

Here we will consider two main types of problems: business problems and scientific problems. Regarding business problems, the initial part of the KDD process is usually identified as *business problem formulation*. We assume that this is done by a human and is formulated in natural language. For instance, if the aim is to obtain an understanding of certain activities of some institution (e.g., company or hospital), we need to define which aspects are of interest. These may include for instance:

1See, e.g., ADS 2019 workshop (Bie et al., 2019).

This information helps to obtain a more detailed understanding of the domain and the data, enabling to focus on the appropriate data mining task.

#### **14.2.2 Generating task descriptors**

The task description is often done in natural language. It needs to be processed in order to extract a set of task descriptors (keywords). Let us see a few examples.

This strategy was exploited, for instance, by Contreras-Ochando et al. (2019), who used certain domain-specific metafeatures to determine the type of task to carry out (e.g., certain metafeatures could indicate that the task involves transforming a date into a normalized format).

Whenever possible, the descriptors used should belong to a given set of common vocabulary/ontology terms, as this facilitates further processing.

#### **14.2.3 Determining the task type and goals**

Particular business or scientific objectives captured by appropriate descriptors determine, to a large extent, what kind of top-level task should be considered. If the aim was to simply obtain understanding of a particular domain, an unsupervised task (e.g., clustering) could be a good choice. If the aim is to predict values of certain variables (objective variables), the most appropriate top-level task could be classification or regression.

Determining the task type from a set of descriptors (keywords) can be formulated as a meta-level classification problem.

We note that a complex task may include various subtasks. The top-level task of a certain type (e.g., classification) may involve subtasks of a different type (e.g., regression).

## **Role of learning goals**

Determining which concepts (descriptors) should be brought into play is influenced by the learning goal. This issue was noted by the Russian psychologist Wygotski (1962). He drew attention to the fact that concepts arise and get developed if there is a specific need for them. Acquisition of concepts is thus a purposeful activity directed towards reaching a specific goal or a solution of a specific task.

This problem has been noted also by people in AI and ML. Various researchers, including Hunter and Ram (1992b,a), Michalski (1994), and Ram and Leake (2005), have argued that it is important to define explicit goals that guide learning. Learning is seen as a search through a knowledge space guided by the learning goal. Learning goals determine which part(s) of prior knowledge are relevant, what knowledge is to be acquired and in what form, how it is to be evaluated, and when to stop learning. The importance of planning in this process has also been identified by Hunter and Ram (1995).

## **Providing a schedule for learning goals**

As some more complex tasks are formulated in terms of multiple goals, it may be desirable to define also an ordering in which (some of) the goals should be achieved. This problem can be seen as the problem of *learning multiple interdependent concepts*. In effect, defining the ordering can be regarded as defining the appropriate procedural bias, as the ordering determines how the hypothesis space should be searched.

## **14.3 Identifying the Task Domain and Knowledge**

The description of a given task using descriptors can be used to determine the domain to which that task belongs. Determining the domain is important, as it facilitates determining what type of data is required to solve the task.

One way to address this is to activate the appropriate domain-specific descriptors (ontology).

This process can be seen as the problem of activating the appropriate domain together with the corresponding set of domain-specific descriptors (ontology). This process can be compared to the process of activating Minsky's frames (Minsky, 1975). However, when the proposal on frames was written, the area of machine learning was not yet very advanced and so the mechanism used that would invoke frames was not well clarified at the time.

Basically, we see two different mechanisms that can be used for this aim. One involves keyword matching, and the other one classification. Both are described in the following subsections.

#### **Identifying the domain by matching descriptors/metafeatures**

Both the task and the domain can be described using descriptors (keywords possibly organized in ontologies) or metafeatures. The domain can then be identified by matching the task and domain descriptors. The aim is to identify the domain with the closest match. It is necessary to define the appropriate measure of similarity between the task descriptors and particular domain descriptors. Naturally, one would seek the domain which is most similar to the given task.

## **Identifying the domain by classification**

Determining the domain can be regarded as a meta-level classification task. The input is a particular set of task descriptors of the given task, and the output is a particular domain characterized by a set of domain descriptors. As domains could be organized in a hierarchy, classification can be done at various levels of this hierarchy. For instance, the task descriptors could suggest that the given task pertains to *medicine* at a higher level and to *obstetrics* at a lower level.

## **Representation of data and goals**

It is important to have an appropriate representation for both data and the goals. Various schemes have been proposed in the past. For instance, Stepp and Michalski (1983) proposed *goal dependency networks* (GDN). In this chapter we have discussed the role of descriptors that help to determine the task type and the appropriate data. Some of these can be identified as descriptors of learning goals. So, for instance, *company profit* can be seen as a descriptor of data, if we are talking about a particular data item stored in a database. If, on the other hand, the aim is to predict its value from other information, then it becomes a learning goal.

## **14.4 Obtaining the Data**

After the domain descriptors have been identified, it is necessary to identify and obtain the actual data. Typically, the aim is to identify a part of the data which is relevant for the task at hand, as this facilitates further processing. If the data were held in a relational database, then it would be necessary to identify the relevant part, i.e., a subset of possibly interconnected tables. Each table has a heading, and the terms used in each column often suggest the type of information contained there. However, sometimes the terms may be abbreviations or codes attributed by the designer. Hence, the problem of matching task descriptors to data descriptors would be facilitated if each name in the table heading were accompanied by the correct ontological descriptor(s).

## **14.4.1 Select existing data or plan how to obtain data?**

The next important question is whether the data is already available or not. If it is available, it is possible to advance. If the data is not available or insufficient for the current task, new data needs to be gathered (e.g., by interacting with the "outside world" or by running experiments). The latter strategy was used in a robot scientist called *Adam* (King et al., 2009). This system autonomously generates hypotheses concerning functional genomics of a particular type of yeast (*Saccharomyces cerevisiae*) and then conducts tests to experimentally verify them.

## **14.4.2 Identifying the domain-specific data and background knowledge**

Identifying the correct part of the data is not a trivial matter. Consider, for instance, the task of predicting whether to concede credit to a given customer. If the data did not include the relevant part (e.g., tables and the corresponding features), the learning

#### 274 14 Automating Data Science

system would be unable to generate the correct inductive hypothesis. In this case, the search space does not include the inductive hypotheses that we would like the system to come up with.

If, on the other hand, the data unnecessarily included too many items, some of which might be irrelevant, the system might not arrive at the right hypothesis within a given time budget. Although in this case the search space would include the right inductive hypotheses, the system may have difficulty discovering them.

In inductive logic programming (ILP) the data is represented in the form of facts. As Srinivasan et al. (1996) have shown, the learning problem can often be defined using a set of constraints. One important one is H ∧ Bk |= E+, stating that the inductive hypothesis H together with the background knowledge Bk should logically imply the set of positive examples E+. 2 Srinivasan et al. (1996) observed that the performance of the learning system is sensitive to the type and amount of background knowledge. In particular, background knowledge that contains information irrelevant to the task considered can make the search for the correct hypothesis more difficult.

So, in general, if our aim was to automate this process, we would need to identify not only the relevant domain-specific data but also the relevant part of background knowledge. More details on some related issues are given in the following subsections.

#### **14.4.3 Obtaining the data and background knowledge from different sources**

As Contreras-Ochando et al. (2019) have pointed out, data science must integrate data from very different data sources. These may include databases, repositories, the web, spreadsheets, text documents, and images, among others. The integration may involve homogeneous sources (e.g., two different databases of similar type, but with different content), or heterogeneous sources (e.g. text and image).

#### **Obtain data by accessing the OLAP data cube**

One useful concept in this area is the so-called *OLAP data cube*, which is simply a multidimensional array of data (Gray et al., 2002). The operation *slice* permits to select a rectangular subset of a cube by choosing a single value for one of its dimensions. The operation *dice* is similar, but allows us to set specific values of multiple dimensions.

Several other operations are discussed in Section 14.5.4, which focuses on rather specific transformations whose aim is to alter the granularity of data.

## **14.5 Automating Data Preprocessing and Transformation**

Data preprocessing and transformation can be regarded as one of many steps in the target workflow (pipeline). We will be discussing various preprocessing operations that are useful in classification pipelines. Usually, the selection of a particular preprocessing operation cannot be done in isolation from other elements in the pipeline. For instance, discretization needs to be carried out only if the particular classifier (e.g., naive Bayes) requires this.

2Srinivasan et al. (1996) refer to this constrain as *strong sufficiency*.

Different methods that can be used to design such a pipeline in a systematic fashion were discussed in detail in Chapter 7 and hence will not be reviewed here again. Our aim here is to complement this discussion by describing studies whose aim is to further automate the process, including tasks such as data wrangling.

As Chu et al. (2016) pointed out, data preprocessing serves either to repair data (e.g., based on some quality rules) or to transform data such that it leads to better results in further analysis (e.g., when applying a machine learning algorithm). Here, we consider the following operations:

Data transformation/data wrangling is discussed in Subsection 14.5.1. Instance selection is discussed in Subsection 14.5.2. Data preprocessing includes various operations, such as:

Detailed descriptions of these operatiopn are discussed in various textbooks (see, e.g., Dasu and Johnson (2003)). Subsection 14.5.3 discusses ways of automatizing the selection of the appropriate preprocessing operation.

Although full automation of data science has been an ultimate goal of many research studies, aiming for this has shown to be computationally expensive, mainly due to the search space involved. In addition, some decisions may require domain knowledge (e.g., is a certain outlier safe to remove or not?), or there may not exist a ground truth to learn from. In Chapter 7, we discussed systems that go a long way towards automating pipeline design, such as Auto-sklearn (Feurer et al., 2015, 2019). This system includes many preprocessing operations (e.g., normalization, missing value imputation, and feature selection). However, even this state-of-the-art system is currently not capable of automating the full data science pipeline. Still, such systems can serve as a useful starting point for further developments.

The aim of this section is to cover other, quite different approaches that typically aim to automate or semi-automate a specific preprocessing technique (e.g., outlier detection). Even though these approaches are not always easy to include in AutoML systems, they do show how to solve sub-problems in isolation, and still help in building larger, efficient systems in the future.

#### **14.5.1 Data transformation/data wrangling**

Data wrangling is the process of transforming and mapping data from one data representation format (e.g., "raw" data) to another. The objective of this transformation is to make it appropriate for subsequent processing. So, for instance, the transformation may convert information in a spreadsheet into a tabular dataset.

## **Inferring data types**

This process often requires inferring the *data types* (e.g., date, float, integer, and string) of certain dataset entities (e.g., features represented in columns). Valera and Ghahramani (2017) proposed a Bayesian method that is able to detect such types based on the distribution of the data in a certain region. However, this does not work optimally if there are missing data and anomalies in the data. System *ptype* by Ceritli et al. (2020) is a more recent method that employs a simpler probabilistic model that detects such anomalies and robustly infers data types.

#### **Some wrangling approaches**

Some systems automatically infer relevant transformations, learned either from a demonstration provided by the user or from a given set of examples. However, as providing examples by the user incurs costs, ideally this is done on the basis of a limited number of examples. This is normally achieved by imposing a strong bias on the actual task (e.g., assumptions about the output format in a particular domain), enabling the system to prioritize some transformations over others.

This was exploited in one early system in this area – *Trifacta Wrangler* (Kandel et al., 2011) – which generates a ranked list of possible transformations. These transformations can be seen as simple programs, which was explored by various researchers already in the 1970s and 1980s (e.g., Mitchell (1977); Mitchell et al. (1983); Brazdil (1981), among others).

#### **System FOOFAH**

Jin et al. (2017) developed a technique to synthesize data transformation programs from example transformations, described using input–output example pairs. Their system, FOOFAH, searches the space of possible data transformation operations to generate a program that will perform the desired transformation. In general, the aim is to transform a poorly structured grid of values (e.g., a spreadsheet table) into a relational table that can be used by ML programs. Such a transformation can be a combination of operators. Some operators are oriented towards columns, and others to rows or even to the whole table (e.g. transpose). Some column operators are shown below:

The authors show that their system obtains the required programs faster than Wrangler discussed earlier.

#### **Usage of ILP in wrangling approaches**

Some researchers followed the approach of inductive programming (IP) or inductive logic programming (ILP) (Gulwani et al., 2015), which led to a wider spread. Microsoft, for instance, has decided to include a data wrangling tool *Flashfill* (Gulwani et al., 2012) in Excel.

The ILP approaches seem particularly useful to the task of transformation, as it is possible to constrain the search by specifying the appropriate domain-specific background knowledge (BK). This was exploited for instance by Contreras-Ochando et al. (2019), who used certain domain-specific metafeatures to determine the type of task to carry out, which in turn brought into play the appropriate background knowledge. The problem tackled by these authors involved recognizing the type of a given *named entity*, which may be a date, email, name, phone number, or some other entity, together with the coding convention that depends on the respective country (e.g. France, UK, etc.). This determines which part of the BK should be activated. In one task, for instance, the goal was to extract the day of the month from inputs in different formats. For instance, "25-03-74" should give 25 and "03/29/86" should return 29.

#### **Systems TDE and SYNTH**

TDE (Transform Data by Example) (He et al., 2018) works in a similar way, but it uses a library of program snippets that represent different transformations. The given examples are used to select the ones that generate the correct output.

SYNTH (De Raedt et al., 2018) can learn to carry out automatic completion of data in a set of worksheets. As the authors have shown, solving this problem requires a number of different steps. First, it is necessary to discover equations and/or obtain one or more predictive models that permit to calculate the values in one cell using the values in other cells. Secondly, it is necessary to apply the equations and/or induced models to fill in empty cells, parts of rows or columns. To address these different tasks, the system uses several different components:

#### **14.5.2 Instance selection and model compression**

Some systems select a subset of data to be cleaned prior to passing them to further processing. *ActiveClean* (Krishnan et al., 2016), for instance, identifies the instances (records) that are more likely to affect the results of subsequent modeling.

The area of outlier detection/elimination is related to this. Many well-known methods exist, such as *isolation forests* (Liu et al., 2012), *one-class SVMs*, or *local outlier factors* (Breunig et al., 2000), among others. Typically these methods do not work well in unsupervised mixed-type scenarios. Eduardo et al. (2020) uses *variational autoencoders* (VAEs) to detect and repair outliers even in this setting.

One application of this is *model compression* (or *model distillation*), where a simpler and hopefully better model is obtained by selecting the training data. One pioneering work in this area was the work of John (1995) on so-called *robust decision trees*, who studied the effects of label noise. After learning a tree, all misclassified instances were removed from the learning set and a new tree was learned. Although this process did not result in an accuracy increase, the resulting tree was much smaller. Similar approaches for k-NN were presented by Tomek (1976) and Wilson and Martinez (2000). Further work on filtering misclassified instances was reported by Smith and Martinez (2018).

#### **14.5.3 Automating the selection of the preprocessing method**

Bilalli et al. (2018, 2019) predicted which preprocessing techniques are appropriate for a given classification algorithm. The preprocessing methods include, for instance, discretization, transformation from nominal to binary, normalization and standardization of values, replacement of missing values, and introduction of principal components. For some of these operations, both supervised and unsupervised variants have been considered. Besides, some of the operations can be applied to just one attribute, while others are global and can thus be applied to all attributes. They build a meta-model in the form of random forest for each target classification algorithm and use it together with the metafeatures of the target dataset to predict which preprocessing technique should be included in the pipeline. The associated probability of classification can be used to order these operations in terms of their expected utility. The resulting system, called PERSIS-TANT, thus recommends which preprocessors to consider given a certain classification algorithm, but not which classifier should be used. Similarly, Schoenfeld et al. (2018) build meta-models predicting when a preprocessing algorithm is likely to improve the accuracy or runtime of a particular classifier.

Other systems extend this idea to *sequences* of preprocessing steps (sub-pipelines). Learn2Clean (Berti-Equille, 2019) attempts to select the optimal sequence of tasks for preprocessing and takes into account the given dataset with the aim of maximizing the performance of the complete workflow. Similarly, in DPD (Quemy, 2019), the sequential model-based optimization (SMBO) technique is used to automatically select and tune preprocessing operators to improve the performance score within a restricted time budget. The authors have found that, for certain NLP preprocessing operators, specific configurations are optimal for several different algorithms.

Finally, some systems produce rankings of promising workflows (pipelines) that can be used to speed up the search. Cachada et al. (2017) and Abdulrahman et al. (2018) describe a method of recommending the most suitable workflow for a target dataset. Unlike PERSISTANT, it is capable of recommending a workflow, which may include preprocessing (CFS feature selection) followed by a classifier with a certain configuration of hyperparameter settings. The recommendations are in the form of a ranking of the most useful workflows. The system can then use this ranking to conduct experiments to identify the workflow with the best performance.

## **14.5.4 Changing the granularity of representation**

Finally, one of the most low-level tasks that could potentially be automated is how to extract data from a database so that it can be used for learning.

#### **Generating aggregate data from an OLAP cube**

As we have mentioned earlier, an *OLAP data cube* is a multi-dimensional array of data (Gray et al., 2002). The operation *drill down/up* allows us to navigate in the data cube, going from the most summarized (up) to the most detailed (down). A *roll-up* involves summarizing the data along a particular dimension. The summarization rule might be an aggregate function, such as computing totals, for one or more of the dimensions, depending on the required level of detail. Common aggregate functions are SUM, AVG, COUNT, MAX, and MIN, among others. The intermediate data structures, sometimes referred to as *cuboids*, can be organized in a lattice. So, the roll-up operation can be seen as moving up in the lattice of cuboids.

Various studies have been carried out in the past. One integrates features obtained as a result of aggregation into the data mining process. For instance, Charnay (2016) used such features in relational decision trees and random forests.

Features that result from aggregation operations are commonly used in many different systems. Let us consider one example here from robotic soccer. As Stone (2000) pointed out, we may want to identify a receiver that can safely receive the ball. One way of finding this is by establishing the distances to surrounding opponent players and identifying the nearest one. The nearest distance represents an aggregate feature.

Some data mining systems, including for instance RapidMiner, provide an aggregate operator, which is well integrated with the rest of the data mining operations (Hofmann and Klinkenberg, 2013).

All legal operations define a search space, which may be too large to search through manually. There are many possibilities regarding which tables may be selected and/or joined. Further choice arises with respect to the selection of columns or aggregation operations that can be applied to them. Hence, there is a need for a kind of automatic data selection that would be able to support this process by exploring this space and providing good recommendation(s) to the user.

The topic of changing the granularity of representation is continued in Chapter 15. However, the focus there is different from this section, as it discusses the problem of changing granularity by introducing new concepts.

## **14.6 Automating Model and Report Generation**

#### **14.6.1 Automating model generation and deployment**

The aim of this step is to a search for the best workflow (pipeline) for the given task. This can be done with a suitable AutoML/meta-level system.

AutoML systems conduct a search for the best workflow. The search may involve experiments on the target dataset with the objective of determining how different candidate workflows (and differently configured variants) perform. Chapter 6 provides more details about the techniques involved and about some systems that exist in this area.

Metalearning systems can be used, provided similar problems have been dealt with in the past and hence a corresponding meta-database exists. Chapters 2, 5, and 7 provide more details. Another possible strategy is to adapt an existing model using knowledge transfer techniques. More details on this can be found in Chapter 12.

#### **14.6.2 Automating report generation**

One good example in this area is the Automated Statistician (Steinruecken et al., 2019). This project aims to automate various phases of data science, including automated construction of models from data, comparison of different models, and automatic elaboration of reports with minimal human intervention. The reports include, besides basic graphs and statistics, human-readable descriptions in natural language.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Automating the Design of Complex Systems**

**Summary.** This chapter discusses the issue of whether it is possible to automate the design of rather complex workflows needed when addressing more complex data science tasks. The focus here is on symbolic approaches, which continue to be relevant. The chapter starts by discussing some more complex operators, including, for instance, conditional operators and operators used in iterative processing. Next, we discuss the issue of introduction of new concepts and the changes of granularity that can be achieved as a result. We review various approaches explored in the past, such as constructive induction, propositionalization, reformulation of rules, among others, but also draw attention to some new advances, such as feature construction in deep NNs. It is foreseeable that in the future both symbolic and subsymbolic approaches will coexist in systems exhibiting a kind of functional symbiosis. There are tasks that cannot be learned in one go, but rather require a sub-division into subtasks, a plan for learning the constituents, and joining the parts together. Some of these subtasks may be interdependent. Some tasks may require an iterative process in the process of learning. This chapter discusses various examples that can stimulate both further research and some practical solutions in this rather challenging area.

## **15.1 Introduction**

The aim of this chapter is to discuss the issue of automating the design of more complex systems than the ones discussed in previous chapters. The aim is to present various techniques which have proved useful in the past research, as they can be regarded as fundamental architectural building blocks of both current and future intelligent systems.

Complex systems can be divided into two major groups. One includes systems that learn from relatively small sets of examples, while the others use large sets of examples (big data) in the learning process. Let us analyze each group in more detail.

As we have mentioned, the systems in the first group typically use modest numbers of examples. Some systems of this kind were discussed in the previous chapter on automating data science and, in particular, data wrangling. These systems often learn from human-provided input, in the form of input–output pairs or even individual steps showing how a particular problem is to be solved. They often exploit symbolic formulations (e.g., ILP) and may exploit background knowledge. This chapter contains material that is closely related to these systems.

The systems in the second group typically use large sets of examples (big data) in the learning process. In this category we find various systems that learn from big data and data streams (see Chapter 11 for details). Many current systems developed for complex tasks, such as vision or machine translation, do not use symbolic methods any more, but deep neural networks (DNNs), embeddings, etc. It is interesting to note that processing the input and constructing embeddings can be seen as a process of capturing the essential information. This knowledge is useful when dealing with new problems. So a question arises of whether symbolic approaches that are discussed in this chapter are at all relevant in the light of current developments.

In our view there are various reasons why they are relevant. One is to draw attention to various common principles that have proved useful in both symbolic and subsymbolic learning. The notion of, for instance, *constructive induction* discussed in Section 15.3 can be related to the introduction of new nodes higher up in the DNN.

The second reason is related to the issue of *explainability*. Many people find it important nowadays that the current systems are able to explain how they arrived at a particular recommendation or decision. As it is difficult to do this in some systems, including NNs, a symbolic model can be maintained to model the more complex system. This way, it is possible to provide the required explanation. If the symbolic model of the underlying complex process is *faithful* in the process of modeling, this solution is more satisfactory than some rather complex explanation that is difficult to comprehend. Explainability contributes towards what some people now call *trusted AI*.

The third reason is that humans use both subsymbolic and symbolic reasoning, and presumably this has a good motive. Possibly it is related to the human capacity called *creativity*. Currently, we can only postulate what this involves, but we believe that abstract concepts play an important role in that process.

It is foreseeable that future systems would benefit from a stronger interplay between the subsymbolic learning (e.g., embeddings) and symbolic counterpart (e.g., ontologies). So, subsymbolic and symbolic approaches could co-exist in a kind of functional symbiosis. Therefore, in our view, an overview of methods and techniques that have proved useful in the process of learning complex systems, albeit symbolic, continues to be relevant.

#### **Overview of this chapter**

More complex tasks may require plans with conditional operators or iterative processing. This issue is briefly reviewed in Section 15.2. Introduction of new concepts is addressed in Section 15.3. There are tasks that cannot be learned in one go, but rather require a sub-division into sub-tasks and then formulating a plan for learning the constituents and joining the parts together. This methodology is discussed in Section 15.4. Some tasks require an iterative process in the process of learning. More details on this can be found in Section 15.5. There are problems whose tasks are interdependent. One such problem is analyzed in Section 15.6.

## **15.2 Exploiting a Richer Set of Operators**