In this study we focus on first-order meta-learning algorithms that aim to learn a parameter initialization of a network which can quickly adapt to new concepts, given a few examples. We investigate two approaches to enhance generalization and speed of learning of such algorithms, particularly expanding on the Reptile (Nichol et al., 2018) algorithm. We introduce a novel regularization technique called meta-step gradient pruning and also investigate the effects of increasing the depth of network architectures in first-order meta-learning. We present an empirical evaluation of both approaches, where we match benchmark few-shot image classification results with 10 times fewer iterations using Mini-ImageNet dataset and with the use of deeper networks, we attain accuracies that surpass the current benchmarks of few-shot image classification using Omniglot dataset. A common drawback consistently seen in traditional machine learning algorithms is the need for large amounts of training data in order to learn a given task BID5 , whereas the ability to grasp new concepts with just a few examples is clearly seen in the way people learn BID6 . This offers many challenges in fast adaption of machine learning in new fields and hence there is a growing interest in algorithms that can learn with limited data availability BID9 .In the development of learning methods that can be trained effectively on sparse data, the process of learning-to-learn is seen as a crucial step BID0 . This is often termed as meta-learning (Schaul & Schmidhuber, 2010) , where a variety of techniques have been presented. In our study, we specifically focus on approaches that learn an initialization of a network, trained on a dataset of tasks. Model-agnostic meta-learning (MAML) BID1 presented this exact approach and its applications of few-shot image classification, where a task was defined as correct classification of a test image out of N object classes, after training on a set of K examples per each class. Furthermore, MAML presented its first-order variant, where the second order derivatives were eliminated during computation while preserving results of the benchmarks. The approach avoided the computational expense of second order derivatives by treating them as constants. Firstorder meta-learning was further investigated in the Reptile algorithm BID7 , where the implementation was simplified eliminating the need for a test set in the tasks. Our study uses Reptile as the algorithm of choice to incorporate the techniques presented to improve generalization of first-order meta-learning.Even though first-order meta-learning has shown to attain fast generalization of concepts given limited data, empirical evaluations on few-shot image classification tasks BID7 show potential to improve the outcomes, especially on inputs with richer features such as real world images. Also drawbacks are seen in slower convergence, requiring a large number of iterations during the training phase. In this study, we investigate techniques used to obtain higher task generalization in models such as regularization BID11 and deeper networks BID3 and we present ways of adapting those in first-order meta-learning.The contributions of our study are as follows.• Introduction of meta-step gradient pruning, a novel approach to regularize parameter updates in first-order meta-learning.• Empirical evaluation of meta-step gradient pruning, achieving benchmark few-shot image classification accuracies with 10 times fewer iterations.• Empirical evaluation of deeper networks in the meta-learning setting, achieving results that surpass the current benchmarks in few-shot image classification. Our proposed novel approach of meta-step gradient pruning demonstrated enhanced generalization effects on the outcomes of first-order meta-learning. The reduced gaps between train and test set accuracies, during training of Omniglot and Mini-ImageNet few-shot classification tasks showed that the parameter initialization has learned to generalize better on the train set.We were able to almost match the benchmark results of first-order MAML and Reptile implementations with 10 times fewer iterations using our algorithm. This further emphasized the improved generalization, helping the parameters to converge the loss on few-shot classification. This increase in speed is vital in tasks such as Mini-Imagenet, because performing first-order meta-learning on real world noisy images is computationally expensive and time-consuming.With our approach of introducing deeper networks to the inner-loop in Omniglot few-shot classification, we showed results surpassing the current benchmarks of both first-order MAML and Reptile algorithms. The expanded parameter space with deeper models shows higher generalization as expected, but it makes the implementation more computationally expensive. This was identified as one drawback of this approach when applying to richer input data such as Mini-ImageNet tasks.Enhanced and fast generalization is utmost important when learning with limited data. Looking forward, we see the importance of elaborated theoretical analysis of meta-step gradient pruning and more techniques of regularization during meta-learning. Also in the future we plan to investigate on the application of first-order meta-learning in other applications such as reinforcement learning.
The study introduces two approaches to enhance generalization of first-order meta-learning and presents empirical evaluation on few-shot image classification. The paper presents an empirical study of the first-order meta-learning Reptile algorithm, investigating a proposed regularization technique and deeper networks