Deep approaches to anomaly detection have recently shown promising results over shallow methods on large and complex datasets. Typically anomaly detection is treated as an unsupervised learning problem. In practice however, one may have---in addition to a large set of unlabeled samples---access to a small pool of labeled samples, e.g. a subset verified by some domain expert as being normal or anomalous. Semi-supervised approaches to anomaly detection aim to utilize such labeled samples, but most proposed methods are limited to merely including labeled normal samples. Only a few methods take advantage of labeled anomalies, with existing deep approaches being domain-specific. In this work we present Deep SAD, an end-to-end deep methodology for general semi-supervised anomaly detection. Using an information-theoretic perspective on anomaly detection, we derive a loss motivated by the idea that the entropy of the latent distribution for normal data should be lower than the entropy of the anomalous distribution. We demonstrate in extensive experiments on MNIST, Fashion-MNIST, and CIFAR-10, along with other anomaly detection benchmark datasets, that our method is on par or outperforms shallow, hybrid, and deep competitors, yielding appreciable performance improvements even when provided with only little labeled data. Anomaly detection (AD) (Chandola et al., 2009; Pimentel et al., 2014) is the task of identifying unusual samples in data. Typically AD methods attempt to learn a "compact" description of the data in an unsupervised manner assuming that most of the samples are normal (i.e., not anomalous). For example, in one-class classification (Moya et al., 1993; Schölkopf et al., 2001 ) the objective is to find a set of small measure which contains most of the data and samples not contained in that set are deemed anomalous. Shallow unsupervised AD methods such as the One-Class SVM (Schölkopf et al., 2001; Tax & Duin, 2004) , Kernel Density Estimation (Parzen, 1962; Kim & Scott, 2012; Vandermeulen & Scott, 2013 ), or Isolation Forest (Liu et al., 2008 often require manual feature engineering to be effective on high-dimensional data and are limited in their scalability to large datasets. These limitations have sparked great interest in developing novel deep approaches to unsupervised AD (Erfani et al., 2016; Zhai et al., 2016; Chen et al., 2017; Ruff et al., 2018; Deecke et al., 2018; Golan & El-Yaniv, 2018; Hendrycks et al., 2019) . Unlike the standard unsupervised AD setting, in many real-world applications one may also have access to some verified (i.e., labeled) normal or anomalous samples in addition to the unlabeled data. Such samples could be hand labeled by a domain expert, for instance. This leads to a semisupervised AD problem: Given n (mostly normal but possibly containing some anomalous contamination) unlabeled samples x 1 , . . . , x n and m labeled samples (x 1 ,ỹ 1 ), . . . , (x m ,ỹ m ), wherẽ y = +1 andỹ = −1 denote normal and anomalous samples respectively, the task is to learn a model that compactly characterizes the "normal class." The term semi-supervised anomaly detection has been used to describe two different AD settings. Most existing "semi-supervised" AD methods, both shallow (Muñoz-Marí et al., 2010; Blanchard et al., 2010; Chandola et al., 2009 ) and deep Akcay et al., 2018; Chalapathy & Chawla, 2019) , only incorporate the use of labeled normal samples but not labeled anomalies, i.e. they are more precisely instances of Learning from Positive (i.e., normal) and Unlabeled Examples (LPUE) (Zhang & Zuo, 2008) . A few works (Wang et al., 2005; Liu & Zheng, 2006; Gör-nitz et al., 2013) have investigated the general semi-supervised AD setting where one also utilizes labeled anomalies, however existing deep approaches are domain or data-type specific (Ergen et al., 2017; Kiran et al., 2018; Min et al., 2018) . Research on deep semi-supervised learning has almost exclusively focused on classification as the downstream task (Kingma et al., 2014; Rasmus et al., 2015; Odena, 2016; Dai et al., 2017; Oliver et al., 2018) . Such semi-supervised classifiers typically assume that similar points are likely to be of the same class, this is known as the cluster assumption (Zhu, 2005; Chapelle et al., 2009 ). This assumption, however, only holds for the "normal class" in AD, but is crucially invalid for the "anomaly class" since anomalies are not necessarily similar to one another. Instead, semi-supervised AD approaches must find a compact description of the normal class while also correctly discriminating the labeled anomalies (Görnitz et al., 2013) . Figure 1 illustrates the differences between various learning paradigms applied to AD on a toy example. We introduce Deep SAD (Deep Semi-supervised Anomaly Detection) in this work, an end-to-end deep method for general semi-supervised AD. Our main contributions are the following: • We introduce an information-theoretic framework for deep AD based on the Infomax principle (Linsker, 1988) . • Using this framework, we derive Deep SAD as a generalization of the unsupervised Deep SVDD method (Ruff et al., 2018) to the general semi-supervised setting. • We conduct extensive experiments in which we establish experimental scenarios for the general semi-supervised AD problem where we also introduce novel baselines. We introduced Deep SAD, a deep method for general semi-supervised anomaly detection. Our method is based on an information-theoretic framework we formulated for deep anomaly detection based on the Infomax principle. This framework can form the basis for rigorous theoretical analyses, e.g. studying the problem under the rate-distortion curve (Alemi et al., 2018) and new methods in the future. Our results suggest that general semi-supervised anomaly detection should always be preferred whenever some labeled information on both normal samples or anomalies is available. ) performing methods in the experimental scenarios (i)-(iii) on the most complex CIFAR-10 dataset. If most points fall above the identity line, this is a very strong indication that the best method indeed significantly outperforms the second best, which often is the case for our Deep SAD method. In this experiment, we examine the detection performance on some well-established AD benchmark datasets (Rayana, 2016) listed in Table 1 . We do this to evaluate the deep against the shallow approaches also on non-image, tabular datasets that are rarely considered in the deep AD literature. For the evaluation, we consider random train-to-test set splits of 60:40 while maintaining the original proportion of anomalies in each set. We then run experiments for 10 seeds with γ l = 0.01 and γ p = 0, i.e. 1% of the training set are labeled anomalies and the unlabeled training data is unpolluted. Since there are no specific different anomaly classes in these datasets, we have k l = 1. We standardize features to have zero mean and unit variance as the only pre-processing step. Table 2 shows the results of the competitive methods. We observe that the shallow kernel methods seem to perform slightly better on the rather small, low-dimensional benchmarks. Deep SAD proves competitive though and the small differences might be explained by the strong advantage we grant the shallow methods in the selection of their hyperparameters. We provide the complete table with the results from all methods in Appendix F for each mini-batch do end for 7: end for Using SGD allows Deep SAD to scale with large datasets as the computational complexity scales linearly in the number of training batches and computations in each batch can be parallelized (e.g., by training on GPUs). Moreover, Deep SAD has low memory complexity as a trained model is fully characterized by the final network parameters W * and no data must be saved or referenced for prediction. Instead, the prediction only requires a forward pass on the network which usually is just a concatenation of simple functions. This enables fast predictions for Deep SAD. Initialization of the network weights W We establish an autoencoder pre-training routine for initialization. That is, we first train an autoencoder that has an encoder with the same architecture as network φ on the reconstruction loss (mean squared error or cross-entropy). After training, we then initialize W with the converged parameters of the encoder. Note that this is in line with the Infomax principle (2) for unsupervised representation learning (Vincent et al., 2008) . Initialization of the center c After initializing the network weights W, we fix the hypersphere center c as the mean of the network representations that we obtain from an initial forward pass on the data (excluding labeled anomalies). We found SGD convergence to be smoother and faster by fixing center c in the neighborhood of the initial data representations as also observed by Ruff et al. (2018) . If sufficiently many labeled normal examples are available, using only those examples for a mean initialization would be another strategy to minimize possible distortions from polluted unlabeled training data. Adding center c as a free optimization variable would allow a trivial "hypersphere collapse" solution for the fully unlabeled setting, i.e. for unsupervised Deep SVDD. Preventing a hypersphere collapse A "hypersphere collapse" describes the trivial solution that neural network φ converges to the constant function φ ≡ c, i.e. the hypersphere collapses to a single point. Ruff et al. (2018) demonstrate theoretical network properties that prevent such a collapse which we adopt for Deep SAD. Most importantly, network φ must have no bias terms and no bounded activation functions. We refer to Ruff et al. (2018) for further details. If there are sufficiently many labeled anomalies available for training, however, hypersphere collapse is not a problem for Deep SAD due to the opposing labeled and unlabeled objectives.
We introduce Deep SAD, a deep method for general semi-supervised anomaly detection that especially takes advantage of labeled anomalies. A new method to find anomaly data, when some labeled anomalies are given, that applies information theory-derived loss based on normal data usuallly having lower entropy than abnormal data. Proposal for an abnormal detection framework under settings where unlabeled data, labeled positive data, and labeled negative data are available, and proposal to approach semi-supervised AD from an information theoretic perspective.