In many real-world settings, a learning model must perform few-shot classification: learn to classify examples from unseen classes using only a few labeled examples per class.
 Additionally, to be safely deployed, it should have the ability to detect out-of-distribution inputs: examples that do not belong to any of the classes.
 While both few-shot classification and out-of-distribution detection are popular topics,
their combination has not been studied. In this work, we propose tasks for out-of-distribution detection in the few-shot setting and establish benchmark datasets, based on four popular few-shot classification datasets.   Then, we propose two new methods for this task and investigate their performance.
 In sum, we establish baseline out-of-distribution detection results using standard metrics on new benchmark datasets and show improved results with our proposed methods. Few-shot learning, at a high-level, is the paradigm of learning where a model is asked to learn about new concepts from only a few examples (Fei-Fei et al., 2006; Lake et al., 2015) . In the case of fewshot classification, a model must classify examples from novel classes, based on only a few labelled examples from each class. The model has to quickly learn (or adapt) a classifier given this very limited amount of learning signal. This paradigm of learning is attractive for the fundamental reason that it resembles how an intelligent system in the real-world has to behave. Unlike the traditional supervised setting, in most real-world settings we would not have access to millions of labelled examples, but would benefit if a few-shot classifier could be deployed, for example, to recognize the facial gestures of a new user, in order to improve human-computer interaction for individuals with motor disabilities (Wang et al., 2019) . For an intelligent system to be deployed in the real-world, not only does it have to do well on the designated task, but perhaps more importantly it should defer its actions when faced with unforeseen situations. In particular, when an input is invalid, or does not belong to any of the target classes, the system should identify the input as out-of-distribution. Successfully detecting out-of-distribution examples is crucial in a safety critical environment. In the supervised setting, out-of-distribution detection has been studied from many different angles (Hendrycks & Gimpel, 2016; Nalisnick et al., 2018) , but this task has not been investigated in the few-shot setting. Worryingly, the current state-of-the-art learning systems, deep neural networks, are known to be unreasonably confident about inputs unrecognizable to humans (Nguyen et al., 2015) , and their predictions can be manipulated with imperceptible changes in input space (Szegedy et al., 2013) . In general, the behavior of deep nets is not well specified when the test queries are out-of-distribution. A standard practice when studying out-of-distribution detection is to evaluate the detection performance when examples from other datasets are mixed into the test set (Hendrycks & Gimpel, 2016) . Here we refer to this type of out-of-distribution input as out-of-dataset (OOS) 1 inputs. In the few-shot setting, within each episode, what is in-distribution is specified based on a few labeled examples, known as the support set. Hence, there naturally exists another type of out-of-distribution input, the inputs that belong to the same dataset but come from classes not represented by the support set. We refer to these as out-of-episode (OOE) examples. These different types of out-of-distribution examples are illustrated in Figure 1 . To the best of our knowledge, this is the first study to investigate both OOS and OOE tasks and report results using commonly-used metrics in the few-shot setting. We showed that existing confidence scores developed in the supervised setting (i.e., setting with a fixed number of classes) are not suitable when used with popular few-shot classifiers. Our proposed confidence scores, -MinDist and LCBO, substantially outperformed the baselines on both tasks across four staple few-shot classification datasets. We hope that our work encourages future studies on quantitative evaluation of out-of-distribution detection and uncertainty in the few-shot setting.
We quantitatively study out-of-distribution detection in few-shot setting, establish baseline results with ProtoNet, MAML, ABML, and improved upon them. The paper proposes two new confidence scores which are more suitable for out-of-distribution detection of few-shot classification and shows that a distance metric-based approach improves performance.