Model-agnostic meta-learning (MAML) is known as a powerful meta-learning method. However, MAML is notorious for being hard to train because of the existence of two learning rates. Therefore, in this paper, we derive the conditions that inner learning rate $\alpha$ and meta-learning rate $\beta$ must satisfy for MAML to converge to minima with some simplifications. We find that the upper bound of $\beta$ depends on $ \alpha$, in contrast to the case of using the normal gradient descent method. Moreover, we show that the threshold of $\beta$ increases as $\alpha$ approaches its own upper bound. This result is verified by experiments on various few-shot tasks and architectures; specifically, we perform sinusoid regression and classification of Omniglot and MiniImagenet datasets with a multilayer perceptron and a convolutional neural network. Based on this outcome, we present a guideline for determining the learning rates: first, search for the largest possible $\alpha$; next, tune $\beta$ based on the chosen value of $\alpha$. A pillar of human intelligence is the ability to learn and adapt to unseen tasks quickly and based on only a limited quantity of data. Although machine learning has achieved remarkable results, many recent models require massive quantities of data and are designed for solving particular tasks. Meta-learning, one of the ways of tackling this problem, tries to develop a model that can adapt to new tasks quickly by learning to learn new concepts from few data points (Schmidhuber, 1987; Thrun & Pratt, 1998) . Among meta-learning algorithms, model-agnostic meta-learning (MAML), a gradient-based metalearning method proposed by Finn et al. (2017) , has recently been extensively studied. For example, MAML is used for continual learning (Finn et al., 2019; Jerfel et al., 2019; Spigler, 2019; Al-Shedivat et al., 2018) , reinforcement learning (Finn et al., 2017; Al-Shedivat et al., 2018; Gupta et al., 2018; Deleu & Bengio, 2018; Liu & Theodorou, 2019) and probablistic inference Yoon et al., 2018; Grant et al., 2018) . The reason why MAML is widely used is because MAML is simple but efficient and applicable to a wide range of tasks independent of model architecture and the loss function. However, MAML is notorious for being hard to train (Antoniou et al., 2019) . One of the reasons why training MAML is hard is the existence of two learning rates in MAML: the inner learning rate α and meta-learning rate β. A learning rate is known to be one of the most important parameters, and tuning this parameter may be challenging even if the simple gradient descent (GD) method is used. Nevertheless, we do not yet know the relationship between these two learning rates and have little guidance on how to tune them. Hence, guidelines for choosing these parameters are urgently needed. In this paper, we investigate the MAML algorithm and propose a guideline for selecting the learning rates. First, in Section 2 we briefly explain by using an approximation how MAML can be regarded as optimization with the negative gradient penalty. Because the gradient norm is related to the shape of the loss surface, a bias towards a larger gradient norm can make training unstable. Next, based on the approximation explained in Section 2, in Section 3, we derive a sufficinent condition of α and β for a simplified MAML to locally converge to local minima from any point in the neighborhood of the local minima. Furthermore, by removing a constraint, we derive a sufficient condition for local convergence with fewer simplifications as well. We find that the upper bound β c of meta-learning rate depends on inner learning rate α. In particular, β c of α ≈ α c is larger than that of α = 0, where α c is the upper bound of α. This is verified by experiments in Section 5. These results imply a guideline for selecting the learning rates: first, search for the largest possible α; next, tune β. We regard a simplified MAML as training with the negative gradient penalty. Based on this formulation, we derived the sufficient condition of the inner learning rate α and the meta-learning rate β for the simplified MAML to locally converge to local minima from any point in the vicinity of the local minima. We showed that the upper bound of β required for the simplified MAML to locally converge to local minima depends on α. Moreover, we found that if α is close to its upper bound α c , the maximum possible meta-learning rate β c is larger than that used while training with ordinary SGD. This finding is validated by experiments, confirming that our theory is applicable in practice. According to this result, we propose a guideline for determining α and β; first, search for α close to α c ; next, tune β based on the selected value of α.
We analyzed the role of two learning rates in model-agnostic meta-learning in convergence. The authors tackled the optimization instability problem in MAML by investigating the two learning rates. This paper studies a method to help tune the two learning rates used in the MAML training algorithm.