Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space. The resulting approach, latent embedding optimization (LEO), decouples the gradient-based adaptation procedure from the underlying high-dimensional space of model parameters. Our evaluation shows that LEO can achieve state-of-the-art performance on the competitive miniImageNet and tieredImageNet few-shot classification tasks. Further analysis indicates LEO is able to capture uncertainty in the data, and can perform adaptation more effectively by optimizing in latent space. Humans have a remarkable ability to quickly grasp new concepts from a very small number of examples or a limited amount of experience, leveraging prior knowledge and context. In contrast, traditional deep learning approaches BID24 BID39 treat each task independently and hence are often data inefficient -despite providing significant performance improvements across the board, such as for image classification BID41 BID14 , reinforcement learning BID29 BID40 , and machine translation BID3 BID44 . Just as humans can efficiently learn new tasks, it is desirable for learning algorithms to quickly adapt to and incorporate new and unseen information.Few-shot learning tasks challenge models to learn a new concept or behaviour with very few examples or limited experience BID6 BID23 . One approach to address this class of problems is meta-learning, a broad family of techniques focused on learning how to learn or to quickly adapt to new information. More specifically, optimization-based meta-learning approaches BID34 BID7 aim to find a single set of model parameters that can be adapted with a few steps of gradient descent to individual tasks. However, using only a few samples (typically 1 or 5) to compute gradients in a high-dimensional parameter space could make generalization difficult, especially under the constraint of a shared starting point for task-specific adaptation.In this work we propose a new approach, named Latent Embedding Optimization (LEO), which learns a low-dimensional latent embedding of model parameters and performs optimization-based meta-learning in this space. Intuitively, the approach provides two advantages. First, the initial parameters for a new task are conditioned on the training data, which enables a task-specific starting point for adaptation. By incorporating a relation network into the encoder, this initialization can better consider the joint relationship between all of the input data. Second, by optimizing in the lower-dimensional latent space, the approach can adapt the behaviour of the model more effectively. Further, by allowing this process to be stochastic, the ambiguities present in the few-shot data regime can be expressed.We demonstrate that LEO achieves state-of-the-art results on both the miniImageNet and tieredImageNet datasets, and run an ablation study and further analysis to show that both conditional parameter generation and optimization in latent space are critical for the success of the method. Source code for our experiments is available at https://github.com/deepmind/leo. We have introduced Latent Embedding Optimization (LEO), a meta-learning technique which uses a parameter generative model to capture the diverse range of parameters useful for a distribution over tasks, and demonstrated a new state-of-the-art result on the challenging 5-way 1-and 5-shot miniImageNet and tieredImageNet classification problems. LEO achieves this by learning a lowdimensional data-dependent latent embedding, and performing gradient-based adaptation in this space, which means that it allows for a task-specific parameter initialization and can perform adaptation more effectively.Future work could focus on replacing the pre-trained feature extractor with one learned jointly through meta-learning, or using LEO for tasks in reinforcement learning or with sequential data.
Latent Embedding Optimization (LEO) is a novel gradient-based meta-learner with state-of-the-art performance on the challenging 5-way 1-shot and 5-shot miniImageNet and tieredImageNet classification tasks. A new meta-learning framework that learns data-dependent latent space, performs fast adaptation in the latent space, is effective for few-shot learning, has task-dependent initialization for adaptation, and works well for multimodal task distribution. This paper proposes a latent embedding optimization method for meta-learning, and claims the contribution is to decouple optimization-based meta-learning techniques from high-dimensional space of model parameters.