This work views neural networks as data generating systems and applies anomalous pattern detection techniques on that data in order to detect when a network is processing a group of anomalous inputs.   Detecting anomalies is a critical component for multiple machine learning problems including detecting the presence of adversarial noise added to inputs. More broadly, this work is a step towards giving neural networks the ability to detect groups of out-of-distribution samples.   This work introduces ``Subset Scanning methods from the anomalous pattern detection domain to the task of detecting anomalous inputs to neural networks.   Subset Scanning allows us to answer the question: "``Which subset of inputs have larger-than-expected activations at which subset of nodes? "  Framing the adversarial detection problem this way allows us to identify systematic patterns in the activation space that span multiple adversarially noised images.   Such images are ``"weird together".   Leveraging this common anomalous pattern, we show increased detection power as the proportion of noised images increases in a test set.    Detection power and accuracy results are provided for targeted adversarial noise added to CIFAR-10 images on a 20-layer ResNet using the Basic Iterative Method attack. The vast majority of data in the world can be thought of as created by unknown, and possibly complex, normal behavior of data generating systems. But what happens when data is generated by an alternative system instead? Fraudulent records, disease outbreaks, cancerous cells on pathology slides, or adversarial noised images are all examples of data that does not come from the original, normal system. These are the interesting data points worth studying. The goal of anomalous pattern detection is to quantify, detect, and characterize the data that are generated under these alternative systems. Furthermore, subset scanning extends these ideas to consider groups of data records that may only appear anomalous when viewed together (as a subset) due to the assumption that they were generated by the same alternative system. Neural networks may be viewed as one of these data generating systems. The activations are a source of high-dimensional data that can be mined to discover anomalous patterns. Mining activation data has implications for interpretable machine learning as well as more objective tasks such as detecting groups of out-of-distribution samples. This paper addresses the question: "Which of the exponentially many subset of inputs (images) have higher-than-expected activations at which of the exponentially many subset of nodes in a hidden layer of a neural network?" We treat this scenario as a search problem with the goal of finding a "high-scoring" subset of images × nodes by efficiently maximizing nonparametric scan statistics in the activation space of neural networks. The primary contribution of this work is to demonstrate that nonparametric scan statistics, efficiently optimized over node-activations × multiple inputs (images), are able to quantify the anomalousness of a subset of those inputs (images) into a real-valued "score". This definition of anomalousness is with respect to a set of clean "background" inputs (images) that are assumed to generate normal or expected patterns in the activation space of the network. Our method measures the deviance between the activations of a subset of inputs (images) under evaluation and the activations generated by the background inputs. The challenging aspect of measuring deviances in the activation space of neural networks is dealing with high-dimensional data, on the order of the number of nodes in a hidden layer × the number of inputs (images) under consideration. Therefore, the measure of anomalousness must be effective in capturing systematic (yet potentially subtle) deviances in a high-dimensional subspace and be computationally tractable. Subset scanning meets both of these requirements (see Section 2). The reward for addressing this difficult problem is an unsupervised, anomalous-input detector that can be applied to any input and to any type of neural network architecture. Neural networks universally rely on their activation space to encode the features of their inputs and therefore quantifying deviations from expected behavior in the activation space has broad appeal and potential beyond detecting anomalous patterns in groups of images. Furthermore, an additional output of subset scanning not fully explored in this paper is the subset of nodes at which the subset of inputs (images) had the higher-than-expected activations. These may be used to characterize the anomalous pattern that is affecting the inputs. The second contribution of this work focuses on detection of targeted adversarial noise added to inputs in order to change the labels to a target class Szegedy et al. (2013) ; Goodfellow et al. (2014) ; . Our critical insight to this problem is the ability to detect the presence of noise (i.e. an anomalous pattern) across multiple images simultaneously. This view is grounded by the idea that targeted attacks will create a subtle, but systematic, anomalous pattern of activations across multiple noised images. Therefore, during a realistic attack on a machine learning system, we expect a subset of the inputs to be anomalous together by sharing higher-than-expected activations at similar nodes. Empirical results show that detection power drastically increases when targeted images compose 8%-10% of the data under evaluation. Detection power is near 1 when the proportion reaches 14%. In summary, this is the first work to apply subset scanning techniques to data generated from neural networks in order to detect anomalous patterns of activations that span multiple inputs (images). To the best of our knowledge, this is the first topic to address adversarial noise detection by considering images as a group rather than individually. The top panel of Table 1 provides detection power for our experiments. Detection power is measured by Area-Under-ROC curves as demonstrated in Figure 2 . The ability to detect targeted noise on individual images varies by class with moderate results and can be viewed as a performance floor. The focus of this work however, is the ability to detect adversarial noise across multiple images simultaneously. To that end, we show how detection power increases as the proportion of noised images increases in the test sets. At a proportion of 10% detection power is higher as a group than it is for an individual image across all target classes. Detection is nearly perfect for all classes at 12% and above. This suggests our scanning method is identifying a subtle anomalous pattern of activations that persists across multiple noised images targeting a single class. We now focus on the "All" category which considers the test sets containing targeted examples from each of the 10 class labels. Detection power lags behind any single target class. This is because in the single target cases, our scanning method is exploiting an anomalous activation pattern that is consistent across multiple images. This pattern is less consistent when targeting different class Table 1 : Detection Power and Accuracy for targeted adversarial noise added to CIFAR-10 images by the Basic Iterative Method attack. Results are provided for detecting individual images and subsets of 500 images where the number of noised images varies from 6% to 14%. labels in the same test set. This suggests that targetted noise is activating the same set of nodes despite the original images coming from different classes. In addition to Detection Power, Table 1 provides precision and recall measurements for the subsets of images identified by our scanning method. Precision is consistently lower than recall. We attribute this to two reasons. The first is that the 500 image test set contains targeted noised examples of a single class label, as well as natural images of that same class. Therefore, we believe the subset of anomalous images is likely to include the noised images and the natural images belonging to the target class, which decreases precision. Another reason for a relatively low precision is due to a static setting of a parameter to the scanning function, α max . For simplicity, this value was set to 0.5 for all runs and may be interpreted as assuming up to half of the data may be affected by the anomalous pattern. This is an inflated value which can be lowered if investigators had an apriori belief on the prevalence of the affected subsets in their data (i.e. the 6%-14% used in our experiments). Lowering this value would almost certainly increase precision (and lower recall). We now consider recall measurements located in the bottom panel of Table 1 . Recall is exceptionally high in our experiments. Similar to the argument for low precision, the high recall values are due to a large, static α max value. A hyper-parameter search is feasible in supervised settings. Instead these experiments were conducted in an unsupervised form with α max set arbitrarily at 0.5. We now highlight a more subtle strength of our method with regards to recall. All things being equal, increasing the number of the noised images should decrease the recall rate as there are more noised images to miss. However, in almost all target classes we observe steady trend or an increase. This demonstrates subset scanning's innate adaptability by maintaining strong recall despite the number of noised images more than doubling (from 6% to 14%). This work uses the Adversarial Noise domain as an effective narrative device to demonstrate that anomalous patterns in the activation space of neural networks can be efficiently quantified and detected across a subset of inputs (images). The primary contribution of this work to the data mining and deep learning literature is a novel, unsupervised anomaly detector that can be applied to any pre-trained, off-the-shelf neural network model. The method is based on subset scanning which treats the detection problem as a search for the highest scoring (most anomalous) subset of node activations × inputs (images) as measured by nonparametric scan statistics. This is the first work to apply subset scanning methods to neural network activations and represents a novel contribution to both domains. Nonparametric scan statistics applied to neural network activations operate on three levels of anomalousness. The first level is at a single activation generated by an input at a node. The anomalousness of this activation is quantified by its empirical p-value that reflects how large this activation is compared to a "background" of activations from known, natural images at the same node. Of course, not every input that has a large activation at this node is anomalous. We therefore must consider the second level measured by NPSS: the anomalousness of a subset of activations for a single image (or equivalently, a single node). This level identifies the most anomalous subset of empirical p-values from a single image (or a single node). Despite the exponentially many subsets to consider, this optimization can be done exactly by only considering a linearly-many number of subsets of activations Neill (2012) . An image (or node) that has a large number of small p−values is considered to be more anomalous. However, the large activations that make one image anomalous may occur at different nodes than an image that is equally anomalous with high activations at a different subet of nodes. This consideration brings us to the third and highest level of anomalousness for NPSS applied to neural network activations: identifying a subset of inputs (images) that have higher-than-expected activations (i.e. large number of low empirical p−values) at a subset of nodes. This search procedure uses the same efficient optimization step to iteratively ascend between identifying the most anomalous subset of nodes (for a given, fixed subset of images) and identifying the most anomalous subset of images (for a given, fixed subset of nodes). In practice, this scanning method is able to identify a high scoring subset of images × nodes from a search space of 500 images and 4096 nodes in 3.8 seconds on average. Efficient optimization is important in the large search space of neural network activations. However, this work also demonstrated that the subset identified by the scanning procedure is relevant for an anomalous pattern of interest. The second contribution of this paper is providing empirical results that subset scanning can detect the presence of targeted adversarial noise. Furthermore the detection power, precision, and recall increase when images with targeted noise are considered together as a group. To the best of our knowledge, this is the first work to consider adversarial noise detection across a group of images. Most adversarial noise defenses only provide detection results for individual images as they fail to scale to detecting at the group level. In practical settings, if a neural network is under a targeted attack there will be systematic differences across the affected images. Our method is capable of detecting these subtle, but systematic, patterns at node activations across multiple images. We also highlight that the adversarial noise detection task was performed completely unsupervised and orthogonal to the original goal of the trained ResNet: to attain high classification accuracy. This suggests that subset scanning over neural network activations will be relevant in a broad range of neural network applications.
We efficiently find a subset of images that have higher than expected activations for some subset of nodes.  These images appear more anomalous and easier to detect when viewed as a group.  The paper proposed a scheme to detect the presence of anomalous inputs based on a "subset scanning" approach to detect anomalous activations in the deep learning network.