# Enhancing Unsupervised Anomaly Detection with Score-Guided Network

Zongyuan Huang, Baohua Zhang, Guoqiang Hu, Longyuan Li, Yanyan Xu∗, Yaohui
Jin∗ Z. Huang, L. Li, Y. Xu, Y. Jin are with the MoE Key Laboratory of
Artificial Intelligence and AI Institute, Shanghai Jiao Tong University,
Shanghai 200240, China. E-mail: {herozen, jeffli, yanyanxu,
<EMAIL_ADDRESS>B. Zhang and G. Hu are with the Big Data and AI Lab of
Industrial and Commercial Bank of China, Shanghai 201206, China. E-mail:
{zhangbh<EMAIL_ADDRESS>
This work has been submitted to the IEEE for possible publication. Copyright
may be transferred without notice, after which this version may no longer be
accessible. ∗Corresponding authors.

###### Abstract

Anomaly detection plays a crucial role in various real-world applications,
including healthcare and finance systems. Owing to the limited number of
anomaly labels in these complex systems, unsupervised anomaly detection
methods have attracted great attention in recent years. Two major challenges
faced by the existing unsupervised methods are: (i) distinguishing between
normal and abnormal data in the transition field, where normal and abnormal
data are highly mixed together; (ii) defining an effective metric to maximize
the gap between normal and abnormal data in a hypothesis space, which is built
by a representation learner. To that end, this work proposes a novel scoring
network with a score-guided regularization to learn and enlarge the anomaly
score disparities between normal and abnormal data. With such score-guided
strategy, the representation learner can gradually learn more informative
representation during the model training stage, especially for the samples in
the transition field. We next propose a score-guided autoencoder (SG-AE),
incorporating the scoring network into an autoencoder framework for anomaly
detection, as well as other three state-of-the-art models, to further
demonstrate the effectiveness and transferability of the design. Extensive
experiments on both synthetic and real-world datasets demonstrate the state-
of-the-art performance of these score-guided models (SGMs).

###### Index Terms:

Anomaly detection, unsupervised learning, scoring network, autoencoder,
regularization.

##  1 Introduction

Anomaly detection aims to identify the samples that considerably deviate from
the expectation of a complex system. There are plentiful applications of
anomaly detection across various domains [1, 2, 3], for example, disease
detection [4], network intrusion detection [5], and financial fraud detection
[6], etc. Existing anomaly detection methods can be grouped into three
categories according to the label availability in the training datasets:
supervised learning methods, semi-supervised learning methods, and
unsupervised learning methods. Supervised and semi-supervised learning methods
rely on labeled data, which rarely exist in some practical applications.
Accordingly, the unsupervised anomaly detection methods have received very
much attention in both academic and industrial communities.

In general, the core idea of the unsupervised methods is to discover and
utilize implicit or extra information to distinguish anomalies. Figure 1
presents the typical pipeline of unsupervised methods for anomaly detection. A
representation learner is first trained to map the original data, e.g.,
tabular, image, or document data, into the desired hypothesis space. Then a
discriminator mostly uses various pre-defined metrics to identify the abnormal
samples. Based on the definition of the discriminator, we can further classify
the unsupervised anomaly detection methods into three categories:
distribution-based methods, distance-based methods, and reconstruction-based
methods [7]. The distribution-based methods identify anomalies by assuming the
normal data follow a certain distribution; the distance-based methods attempt
to enhance the distance between the abnormal and normal data in a latent
space; the reconstruction-based methods identify anomalies by the similarity
between original data and reconstructed data. The final identification of
anomaly always solely relies on the pre-defined distance calculated between
the original samples or their representations in these existing methods, as
illustrated in Figure 1.

Most existing unsupervised methods face the following challenges: (1) Clean
data with only normal samples are required to train their models [8, 9, 10,
11, 12]. In these cases, the contaminated data with both normal and abnormal
samples, which commonly exist in real-world applications, will hinder the
models from fitting normal data, resulting in performance degradation. (2) The
manually selected metric to compute the anomaly scores will impact the
performance and can hardly ”one-model-fits-all” [13, 10, 14]. (3) Data
distribution can be divided into three fields: obvious-normal field,
transition field, and obvious-abnormal field, as shown in Figure 2. The
samples in obvious-normal or obvious-abnormal fields can be clearly
distinguished as normal or abnormal data. However, the mixture of normal and
abnormal data in the transition field hampers the identification of the
abnormal samples.

Figure 1: The typical pipeline of unsupervised anomaly detection and the
proposed scoring network. Different from previous unsupervised anomaly
detection methods that calculate anomaly scores with a pre-defined metric, our
scoring network learns the metrics from data to discriminate anomalies and
enlarges the anomaly score disparity. The proposed scoring network can be
embedded into existing methods without additional assumptions. The
representation learner is simultaneously trained with the scoring network.
That is, the scoring network guides the representation learner to learn the
information with better discrimination ability from the input data. Figure 2:
A toy example to illustrate the concepts of obvious-normal, obvious-abnormal,
and transition fields.

To overcome these challenges, we propose a scoring network to promote the
performance of anomaly identification via learning and enhancing the disparity
between the normal and abnormal data. As shown in Figure 1, the scoring
network is connected with the representation learner, which maps the original
data into a hypothesis space. The scoring network guides the training of the
representation learner to extract valuable information, which is more helpful
for discrimination, from the data in the obvious-normal and obvious-abnormal
fields. Specifically, we introduce a score-guided regularization in the
scoring network to assign smaller scores to the obvious-normal data and larger
scores to the obvious-abnormal data, making full use of the obvious data and
enhancing the optimization of the representation learner. Thus the normal and
abnormal data in the transition field will be guided to different directions,
and the difference is expected to gradually increase during the model
training. Moreover, the scoring network is devised to directly learn the
discrimination metrics and output the anomaly scores in an end-to-end fashion,
instead of relying on pre-defined distance metrics or reconstruction errors,
thereby can be flexibly embedded into most existing anomaly detection models.

We further propose a novel instantiation, embedding the scoring network into
an autoencoder framework, named score-guided autoencoder (SG-AE). The
autoencoder reconstructs data instances and learns their representations in a
latent space. Then, the scoring network assigns scores to these
representations and attempt to maximize the disparity of the normal and
abnormal samples. A sample with a higher anomaly score is more likely to be an
anomaly. To explain how the scoring network works, we conduct simulation
experiments to observe the changes of the score differences in the transition
field. Also, we compare the performances of SG-AE with classic and state-of-
the-art methods on seven tabular datasets. We also present the distributions
of anomaly scores of normal and abnormal data and utilize t-SNE to reveal the
anomaly detection performance in a 2D space. Moreover, we extend the scoring
network to three state-of-the-art models and explore the performance
improvement of these score-guided models (SG-Models). We also examine the
potential of using scoring network on two document datasets and one image
dataset. The main contributions of this work are summarized as follows:

  * •

We propose a scoring network with score-guided regularization, which utilizes
the obvious data to strengthen the representation learner and expand the
scores disparity between normal and abnormal samples in the transition field,
thus enhance the anomaly detection ability. The scoring network can directly
learn the anomaly scores and be easily integrated into most unsupervised
models without imposing more assumptions.

  * •

We integrate the scoring network into an autoencoder structure and introduce a
simple but effective instantiation, SG-AE. SG-AE breaks the limitation of the
autoencoder on normal data input and achieves competitive performance on
several datasets. We also incorporate the scoring network to three state-of-
the-art models to examine the adaptability to different methods.

  * •

Extensive experiments on synthetic and real-world datasets demonstrate the
effectiveness of the scoring network. The SG-AE outperforms the classic and
state-of-the-art methods and the other SG-Models show performance improvements
with the help of scoring network. Moreover, our scoring network is robust to
the anomaly rate by learning the disparity between normal and abnormal data,
which is also demonstrated by the experiments.

##  2 Related Work

In this section, we briefly review the related work in unsupervised anomaly
detection and revisit the use of regularization techniques in anomaly
detection. Then we compare our design with the existing methods.

###  2.1 Unsupervised Anomaly Detection

Distribution-based methods. Distribution-based methods assume that normal data
follow a certain distribution in the original or latent space and anomalies
deviate from the distribution [15, 16, 17]. The classic distribution-based
methods, based on extreme-value analysis, consider the tail of a probability
distribution as anomalies [18]. In recent years, deep learning brings more
possibilities with more distribution hypotheses in the hidden space. For
instance, Deep Autoencoding Gaussian Mixture Model (DAGMM) [10] assumes that
the low-dimensional representations and reconstruction errors of normal data
follow a Gaussian Mixture Model, and uses the probability value to distinguish
between normal and abnormal data. Some one-class classification methods also
fall into this category due to the consideration of manifold distribution.
Deep SVDD [9] attempts to map the normal data into a hypersphere in a latent
space and the abnormal data fall outside the hypersphere. DROCC [11] assumes
that the normal data lie on a low-dimensional manifold and distinguishes data
via Euclidean distance. It generates synthetic abnormal data to learn a robust
boundary with a well-designed optimization algorithm. GOAD [12] maps data into
a sphere feature space with random affine transformations, and anomalies are
assumed to deviate from the center. Some methods, like, Outlier Exposure [19],
utilize extra information provided by auxiliary datasets to learn the normal
distribution and detect the out-of-distribution samples. Note that although a
number of generative methods assume various distributions in latent spaces,
they finally determine anomalies based on the reconstruction errors. As such,
we classify them into reconstruction-based methods.

Distance-based methods. Distance-based methods consider the positional
relations or neighbor structures in the original or hidden space, assuming
that anomalous samples stay far away from the normal ones [20, 21]. Nearest-
neighbor-based methods, such as LOF [22] and DN2 [23], are typical of the
distance-based methods. Likewise, the tree-based methods, like Isolation
Forest (iForest) [24] and RRCT [25], can be regarded as distance-based methods
since they attempt to capture high-density fields that are reflected in the
depth or complexity of the tree [7]. Deep learning technologies are
advantageous tools to learn data representation so that distance can be
measured in a hypothesis space [26]. For example, Random Distance Prediction
(RDP) [27] builds deep neural networks to learn a certain random mapping,
which can preserve the proximity information for data.

Reconstruction-based methods. Reconstruction-based methods train models to
reconstruct data, with an assumption that the trained models learn the
patterns of the major data and anomalies are unable to be well reconstructed.
Autoencoder framework [28] is one fundamental architecture in reconstruction-
based methods. Robust Deep Autoencoder (RDA) [13] is an early work to build an
autoencoder on corrupted data. It isolates noise and outliers with robust
principal component analysis, and thereafter trains an autoencoder. Robust
Subspace Recovery Autoencoder (RSRAE) [14] constructs a robust subspace
recovery layer within an autoencoder with a well-designed loss. Cosine
similarity between original and reconstructed data is used to identify the
abnormal samples. CoRA [29] is a transductive semi-supervised model, which
modifies the autoencoder framework with one encoder and two decoders. The two
decoders distinguish normal data from abnormal data by reconstruction errors
and then decode them separately. Anomaly detection with generative models can
also be considered as reconstruction-based methods. VAE adopts reconstruction
probability to identify anomalies [30, 31, 3] while GAN-based methods detect
anomalies by the discriminator due to the inaccurate reconstruction by the
generator [32, 33, 34].

###  2.2 Anomaly Detection with Regularization

Regularization is a widely used technique to alleviate the over-fitting
problem [35] and achieve adversarial robustness [36]. Many pieces of work
utilize regularization to encourage autoencoder to learn an informative low
dimensional representation of input data [37, 38, 39, 40]. In the context of
anomaly detection, regularization methods are introduced to enforce robustness
against anomalies, encouraging the model to learn key underlying regularities
[41]. Motivated by robust principal component analysis (RPCA), RDA [13] adds
sparsity penalty $\ell_{1}$ and group penalty $\ell_{2,1}$ into its RPCA-like
loss function of auto-encoder to improve robustness. RSRAE [14] further
incorporates a special regularizer into the embedding of the encoder to
enforce anomaly-robust linear structure. DAGMM [10] introduces $\ell_{1}$
penalty to constrain training. Some one-class classification methods also
employ the regularization to better learn the boundary and improve
performances. Ruff et al. presented a unified view of anomaly detection
methods and comprehensively summarized the methods using regularizations [7].

###  2.3 Comparison with existing methods

Comparing to the existing work, the proposed method has the following
characteristics to tackle the challenges mentioned in Section 1: (1) Many
existing unsupervised models are only trained with normal samples [8, 9, 10,
11, 12]. Although some methods can deal with the contaminated data [13, 27,
14], they make efforts to retain the obvious-normal data and filter out the
suspected abnormal data. Our method leverages both normal and abnormal samples
in the datasets and attempts to enhance the detection capabilities in the
transition field. Specifically, our score-guided regularization utilizes the
obvious data to enlarge the score disparity between normal and abnormal data,
and guide the training of the representation learner and the scoring network.
By doing so, the proposed scoring network can improve the ability of the
representation learner and the robustness of the entire detection method. (2)
The existing unsupervised models define anomaly scores with selected
appropriate metrics, such as Euclidean distance for distance-based methods [9,
26, 11] or cosine similarity for reconstruction-based methods [13, 32, 14].
The anomaly scores are not directly optimized in an end-to-end fashion. Devnet
[42] is the first weakly supervised method to achieve end-to-end anomaly score
learning. However, Devnet guides model training with labels that are missing
in the unsupervised settings and treats the unknown anomalies as normal data.
In contrast, our scoring network can be incorporated into existing
unsupervised methods without additional assumptions, expanding their ability
to handle contaminated data sets and directly optimize anomaly scores.

##  3 Methodology

In this section, we first introduce the score-guided regularization and then
incorporate the scoring network into different unsupervised methods. These
score-guided models are collectively called SG-Models (SGMs). Finally, the
algorithm of SGMs is elaborated.

###  3.1 Score-guided Regularization

Let $\mathbf{X}=\\{\bm{x}_{i}\\}_{N}$ denote a dataset with $N$ samples where
$\bm{x}_{i}\in\mathbb{R}^{D}$. The common practice of unsupervised anomaly
detection methods is to choose one hypothesis mentioned in Section 2 to build
a model. As shown in Figure 1, after the representation learner maps the data
samples to a latent space, a pre-defined function $f(\cdot)$ need to be
selected to identify the anomalous data in the latent space. According to the
chosen hypothesis, $f(\cdot)$ can be distribution functions $f(\bm{z})$,
reconstruction errors $f(\bm{x},\bm{\tilde{x}})$, or distance relations
$f(\bm{z}_{i},\bm{z}_{j})$, as:

$f(\cdot)=\begin{cases}f(\bm{z}),&\text{distribution-based}\\\
f(\bm{x},\bm{\tilde{x}}),&\text{reconstruction-based}\\\
f(\bm{z}_{i},\bm{z}_{j}),&\text{distance-based}\end{cases},$ (1)

where $\bm{z}$ is the learned data representation, $\bm{\tilde{x}}$ is the
reconstructed counterpart of $\bm{x}$.

The function values of anomalies deviate from the major data, and a larger
value suggests a higher probability of being an anomaly. However, selecting or
designing an effective $f(\cdot)$ requires prior domain knowledge about the
tasks and there is almost no universal metric to fit various datasets.
Therefore, an intuitive practice is to directly learn the metrics through a
neural network and enlarge the deviation between normal and abnormal data to
improve the detection capability. However, the prerequisite for the correct
expansion of the score disparity is that we can distinguish between the normal
and abnormal data, which seems to be a paradox. It is not a simple task for
the scoring network to assign correct scores to samples in the unsupervised
settings, especially to the samples in the transition field.

In fact, we can use the discrimination metric $f(\cdot)$ as a self-supervised
signal to guide the learning of anomaly score. Supposing that the metric has
the ability to detect the obvious-normal and obvious-abnormal data, the
scoring network are expected to guide the score distribution of the obvious
data first and then gradually try to guide the non-obvious data in the
transition field. Specifically, these obvious data are effortless to identify,
thus their anomaly scores can be guided faster to converge to the expected
values. The representation learner extracts more valuable information from the
obvious data, and the scoring network can more efficiently capture differences
from data representations. Although these non-obvious data are difficult to
distinguish and the anomaly scores of them might even be guided in the wrong
direction in the early stage of training, the whole model will gradually be
optimized and be capable of identifying the correct guidance direction.
Finally, these non-obvious data are expected to be better distinguished.

To achieve this, we propose a scoring network with a score-guided
regularization to learn and enlarge the anomaly score disparity and to enhance
the capability of the representation learner, as shown in Figure 1. Supposing
$\bm{z}$ is the representation of a data sample in the latent space, the
scoring network is devised to take $\bm{z}$ as input and output an anomaly
score in an end-to-end manner. As expected to assign smaller anomaly scores to
obvious-normal samples and larger scores to obvious-abnormal samples, the
regularization function $L_{\rm{SE}}$ can be defined as follows:

$L_{\rm{SE}}(\cdot,s)=\begin{cases}|s|,&f(\cdot)<\epsilon_{1}\\\
\lambda_{a}\cdot\max(0,a-s),&f(\cdot)>\epsilon_{2}\end{cases},$ (2)

where $f(\cdot)$ is the aforementioned discrimination function, $s$ is the
learned anomaly score, and $\lambda_{a}$ is a weight parameter to adjust the
effect of score guidance for anomalies. $\epsilon_{1}$ and $\epsilon_{2}$ are
thresholds to divide obvious-normal and obvious-abnormal fields, as shown in
Figure 2. As the training progresses, the anomaly scores of obvious-normal
samples are expected to be zero and the scores of obvious-abnormal samples are
expected to be large than value $a$. However, the selection of the two
thresholds is difficult in unsupervised settings. Considering that most
samples in a dataset are normal, we apply only one threshold $\epsilon$ to
divide the obvious-normal field and the suspected abnormal field. Then the
regularization function $L_{\rm{SE}}$ is revised as follows:

$L_{\rm{SE}}(\cdot,s)=\begin{cases}|s-\mu_{0}|,&f(\cdot)<\epsilon\\\
\lambda_{a}\cdot\max(0,a-s),&f(\cdot)\geq\epsilon\end{cases}.$ (3)

We set a very small positive value $\mu_{0}$ approaching zero to be the target
anomaly scores for the obvious-normal data, due to that the zero scores will
force most weights of the scoring network to be $0$. Although the normal data
in the transition field is regarded as the suspected abnormal data, they will
be gradually separated from the abnormal data with the knowledge learned from
the obvious-normal data. From another perspective, the regularization function
can be rewritten as follows:

$L_{\rm{SE}}=L_{normal}+\lambda_{a}\cdot L_{abnormal},$ (4)

where $L_{normal}$ is the the normal data part and $L_{abnormal}$ is the
suspected abnormal data part.

The score-guided regularization is not constrained by new assumptions, so it
can be integrated with existing unsupervised methods. Let $L_{\rm{OE}}$ be the
loss function of an existing unsupervised model, the total loss function $L$
of the integrated model with score-guided regularization is

$L=L_{\rm{OE}}+\lambda_{\rm{SE}}\cdot L_{\rm{SE}},$ (5)

where $\rm{\lambda_{SE}}$ is a hyperparameter to adjust the effect of score-
guided regularization.

Paremeter analysis. The score-guided regularization introduces four
parameters: $\lambda_{\rm{SE}}$, $\lambda_{a}$, $\epsilon$ and $a$, which
seems to make it difficult to determine the appropriate parameter values in
real-life applications. Here we analyze the real impact of these parameters.
The parameter $\lambda_{\rm{SE}}$ adjusts the effect of the score-guided
regularization to the origin loss and $\lambda_{a}$ adjusts the effect of the
suspected abnormal part. The real impact of the two $\lambda$ can rewrite the
Eq. (5) as:

$L=L_{\rm{OE}}+\lambda_{normal}\cdot L_{normal}+\lambda_{abnormal}\cdot
L_{abnormal},$ (6)

where $\lambda_{normal}$ and $\lambda_{abnormal}$ adjust the effect of the
normal data part and the suspected abnormal data part. The parameter
$\lambda_{normal}$ and $\lambda_{abnormal}$ have different value ranges but
have the same meaning and influence as the parameter $\lambda_{\rm{SE}}$ and
$\lambda_{a}$. When we incorporate the scoring network with the autoencoder as
mentioned in the next subsection, the $L_{\rm{OE}}$ in Eq. (6) changes to the
reconstruction error $L_{\rm{RE}}$, and the total loss function has a similar
form with DAGMM [10] and RSRAE [14], both of which also have two $lambda$ to
adjust the regularization effect. The parameter $a$ determines the guided-
position for anomaly scores. Intuitively, a large $a$ can better enlarge the
score gap between the normal and abnormal data. However, a too-large $a$ will
only lengthen the distribution of scores, and the performance improvement will
stabilize with the increase of $a$. Because the anomaly score is learned by
the scoring network and $a$ determines the effect of the regularizer together
with $\lambda_{a}$, the actual impact of $a$ is small, not requiring
adjustment, therefore we set $a=6$ in all experiments. The parameter
$\epsilon$ determines the rate of normal and suspected abnormal data. For
convenience, we can select a percentile, denoted as $\epsilon_{p}$, of the
values of $f(\cdot)$, to find the corresponding $\epsilon$. The parameter
$\epsilon_{p}$ can be smaller than the normal data ratio, because the scoring
network mainly use the obvious-normal data to guide the score distribution in
transition field, which has been confirmed in experiments. Notably, the
distribution of anomaly scores can be a signal to reflect the model
performance and help parameter selection since the anomaly scores are directly
learned and guided by the scoring network. We can also design an early stop
mechanism based on the constraint of anomaly scores.

The score-guided regularizer is easy to combine limited labels by using labels
instead of $f(\cdot)$ for the labelled data. In this case, the regularizer
does not rely entirely on labels to guide the anomaly scores like DevNet, due
to the use of self-supervised signals.

###  3.2 Score-guided Models

As mentioned above, we can integrate the scoring network into unsupervised
anomaly detection methods. The score-guided methods can be collectively called
SG-Models (SGMs). Here, we first propose SG-AE, an instantiation of applying
scoring network to one reconstruction-based method, and then introduce a
general form of SGMs.

As illustrated in Figure 3, SG-AE consists of a reconstructor and a score
guider. Given a data sample $\bm{x}$, the reconstructor first maps it to a
representation $\bm{z}$ in a latent space and then generate an estimation
$\bm{\tilde{x}}$. The score guider learns an anomaly score $\bm{s}$ using
$\bm{z}$ as its input. In practice, we adopt an autoencoder framework as our
reconstructor and a fully connected network in score guider. For tabular data,
we also use fully connected networks in the autoencoder. It’s noteworthy that
this autoencoder network is easy to transfer to other data types. For example,
we can utilize recurrent networks for sequence data and convolutional networks
for image data [42, 14].

Figure 3: The proposed score-guided autoencoder (SG-AE).

Let $\mathcal{E}$ and $\mathcal{D}$ denote the encoder and decoder in an
autoencoder, respectively. We denote the score networking as $\mathcal{S}$,
and thus have

$\bm{z}=\mathcal{E}(\bm{x}),\quad\bm{\tilde{x}}=\mathcal{D}(\bm{z}),\quad\bm{s}=\mathcal{S}(\bm{z}),$
(7)

where $\bm{x}$ is the input data; $\bm{\tilde{x}}$ is the reconstructed
counterpart of $\bm{x}$; $\bm{z}$ is the latent representation; and $\bm{s}$
is the anomaly score.

The end-to-end network SG-AE can be represented as $\mathcal{M}$:

$\mathcal{M}(\bm{x})=\mathcal{S}(\mathcal{E}(\bm{x})).$ (8)

We can obtain the parameters of $\mathcal{M}$ by minimizing a loss function,
which is a combination of two parts: the reconstruction loss and the score-
guidance loss. For the autoencoder, we use the $\mathcal{L}_{2}$ loss to
assess the reconstruction,

$L_{\rm{RE}}(\bm{x},\bm{\tilde{x}})=\frac{1}{N}\sum_{i=1}^{N}\big{\|}\bm{x}_{i}-\bm{\tilde{x}}_{i}\big{\|}_{2}.$
(9)

In the reconstruction-based settings, we also utilize the reconstruction error
as a self-supervised signal, that is, we use $\mathcal{L}_{2}$ as $f(\cdot)$
in Eq. (3). Thus, we can rewrite the score-guided regularization in (3) and
define the score-guidance loss as,

$\leavevmode\resizebox{368.57964pt}{}{$L_{\rm{SE}}(\bm{x}_{i},\bm{\tilde{x}}_{i},s_{i})=\begin{cases}|s-\mu_{0}|,&\big{\|}\bm{x}_{i}-\bm{\tilde{x}}_{i}\big{\|}_{2}<\epsilon\\\
\lambda_{a}\cdot\max(0,a-s),&\big{\|}\bm{x}_{i}-\bm{\tilde{x}}_{i}\big{\|}_{2}\geq\epsilon\end{cases}$}.$
(10)

We finally define the overall loss function of SG-AE as the combination of the
two loss items in Eq. (9) and (10), that is,

$L(\bm{x},\bm{\tilde{x}},s)=L_{\rm{RE}}(\bm{x},\bm{\tilde{x}})+\lambda_{\rm{SE}}\cdot\frac{1}{N}\sum_{i=1}^{N}L_{\rm{SE}}(\bm{x}_{i},\bm{\tilde{x}}_{i},\bm{s_{i}}).$
(11)

Our goal is to minimize the loss function (11). Lower reconstruction loss in
Eq. (9) ensures better reconstructions $\bm{\tilde{x}}$ and representations
$\bm{z}$ of normal data. As the score-guidance loss decreases, the anomaly
scores of normal data approach zero and the scores of suspected abnormal data
tend to $a$, thus the score disparities between normal and abnormal data
continue to widen. A higher anomaly score of a sample indicates higher
possibility to be a anomaly. By thresholding the anomaly scores, one can
distinguish between normal and abnormal data samples.

In addition to SG-AE with reconstruction-based assumption, the scoring network
can also be applied to other unsupervised methods, such as distance-based
methods and distribution-based methods. Here, we discuss a general form of
combining the scoring network with unsupervised methods. Let $\mathcal{R}$ be
a representation learner in an unsupervised method. With the discrimination
metric $f(\cdot)$, the general form of an unsupervised method is:

$\mathcal{M}(\bm{x})=f(\mathcal{R}(\bm{x})).$ (12)

Combined with the scoring network $\mathcal{S}$, the general form of SGMs is:

$\mathcal{M}(\bm{x})=\mathcal{S}(f,\mathcal{R}(\bm{x})).$ (13)

By optimizing the total loss function 5, the score-guided regularization can
encourage the SGMs learn and guide the anomaly scores, and also help the
representation learner of the original model to learn better data
representations. To better understand the scalability, we extend the scoring
network to three state-of-the-art models, DAGMM [10], RDA [13] and RSRAE [14],
and evaluate the performance in the experiments.

###  3.3 The SGM Algorithm

We illustrate the training process of SGMs in Algorithm 1. The parameters of a
SGM are initialized randomly and optimized in the training iteration (Steps
2-10) to minimize the loss in Eq. (5). Specifically, the data representation
$\bm{z}$ is learned by the representation learner $\mathcal{R}$, through
$\bm{z}=\mathcal{E}(\bm{x})$ in Step 5. Then the scoring network learns the
anomaly scores $\bm{s}$ through $\bm{s}=\mathcal{S}(\bm{z})$ in Step 6. Then
Step 7 calculates the loss function (5) and Step 8 takes a backpropagation
algorithm and updates the network paramereters. Data representation and
anomaly score distribution are optimized during training. Given a trained SGM,
anomaly scores can be directly calculated given new data samples.

Algorithm 1 Training SGMs

0: Training set $\\{\bm{x}_{i}\\}_{N}$, $\bm{x}_{i}\in\mathbb{R}^{D}$

0: SGM $\mathcal{M}$:An optimized anomaly detection network

1: Randomly initialize the network parameters of $\mathcal{R}$, $\mathcal{S}$

2: for each epoch do

3: Divide input data into batches

4: for each batch do

5: $\bm{z}\leftarrow\bm{z}=\mathcal{R}(\bm{x})$

6: $\bm{s}\leftarrow\bm{s}=\mathcal{S}(\bm{z})$

7: Calculate loss function $L$

8: Backpropagate and update the parameters of $\mathcal{R}$, $\mathcal{S}$

9: end for

10: end for

11: return Optimized SGM

##  4 Experiments

In this section, we empirically evaluate the effectiveness and robustness of
the SGMs on both synthetic and real-world datasets.

###  4.1 Simulation Experiments

The intention of the score-guidance design is to enlarge the disparity between
the normal and abnormal samples, especially for those fall into the transition
fields, during the training process. To verify that, we conduct simulation
experiments to observe the change of anomaly scores during the model training.
We compare our SG-AE model with AutoEncoder (AE) [28] and RDP [27] in three
artificial datasets.

Data generation. As illustrated in Figure 4, we generate a one-dimensional
dataset and two two-dimensional datasets. The normal data are shown in blue
while the abnormal data are in red. We utilize different equations to generate
datasets of different complexity. For the one-dimensional scenario, we sample
data points from Gaussian distribution $\mathcal{N}(\mu,\sigma^{2})$ in
geometric coordinates. For the two-dimensional scenarios, we generate data via
controlling two variables in the polar coordinates and then transform them
into geometric coordinates. One variable follows a Gaussian distribution and
the other follows a uniform distribution. The normal and abnormal data are
sampled with different mean $\mu$ and same variance $\sigma^{2}$. The detailed
equations are shown in Figure 4.

Settings. The training and testing sets are independently generated, each with
10,000 data samples. The ratio of the abnormal to normal data is one to nine.
In the testing sets, we use three values to divide each dataset into four
parts $({\rm{R}}_{0}{\sim}{\rm{R}}_{3})$, as shown in Figure 4. Based on the
3-$\sigma$ rule, the thresholds are chosen with Gaussian parameters $\mu$ and
$\sigma$: $\\{\mu+\sigma,\mu+2\sigma,\mu+3\sigma\\}$. Most of the data in
${\rm{R}}_{0}$ are normal, namely obvious-normal. Similarly, ${\rm{R}}_{3}$ is
regarded as obvious-abnormal; ${\rm{R}}_{1}$ and ${\rm{R}}_{2}$ are two
transition fields. In each field, we calculate the difference between the
average anomaly scores of abnormal data and normal data. That is,

$S_{i}=\frac{1}{N_{ai}}\sum_{j=1}^{N_{ai}}s_{j}-\frac{1}{N_{ni}}\sum_{k=1}^{N_{ni}}s_{k},i\in[0,1,2,3],$
(14)

where $S_{i}$ is the score difference in ${\rm{R}}_{i}$, $N_{ai}$ and $N_{ni}$
are the numbers of abnormal and normal samples in ${\rm{R}}_{i}$, $s_{j}$ is
the anomaly score of a sample. The score differences reflect the ability to
distinguish between normal and abnormal data. We then use the Area Under
Receiver Operating Characteristic Curve (AUC-ROC) to evaluate the model
performances. The AE and SG-AE share the same Adam optimizer, learning rate,
and fully connected networks. The layer sizes are set as 20 for the one-
dimensional dataset and (64, 20) for the two-dimensional datasets. The scoring
network in SG-AE is also a fully connected network with layer size (20, 1).
The RDP keeps the originally proposed settings. The batch size and the
training epoch are 1024 and 100, respectively.

Figure 4: Simulation experiments in three synthetic datasets.
$({\rm{R}}_{0}{\sim}{\rm{R}}_{3})$ represent fields with different ratio of
anomaly data. For SG-AE, the score difference between normal and abnormal data
in each field gradually expands with the training process.

Results analysis. As shown in Figure 4, for AE, the score differences decrease
during training, suggesting that AE attempts to reconstruct the abnormal data.
The continuous decrease of score differences $S_{1}$ and $S_{2}$ mean that AE
starts to fail in distinguishing the normal and abnormal data in the
transition field. For RDP, it uses a boosting process to filter anomalies and
the iteration size is set as 8, thus the curves of score difference are jagged
and the total epochs are 800. The changes of $S_{1}$ and $S_{2}$ are not
obvious. For SG-AE, the score difference in each field is increasing during
training, suggesting that the model reacts differently to normal and abnormal
data. Specifically, the abnormal data are guided to obtain a larger anomaly
score, while the normal data to a smaller anomaly score. More important, as
expected, the increase of $S_{1}$ and $S_{2}$ indicates the improvement of the
detection ability of the model in the transition field. Moreover, the SG-AE
achieves the best AUC-ROC performances in the three datasets. The results
demonstrate the effectiveness of the scoring network.

###  4.2 Evaluation Experiments

####  4.2.1 Datasets Description

As shown in Table I, we use seven publicly available tabular datasets in our
experiments. Two datasets are in the healthcare domain, e.g., diagnosis of
breast cancer (bcsc) [43] and diabetic detection (diabetic) [44]. Another two
datasets are related to attack in cybersecurity, e.g., intrusion [45] and
attack [46]; and Market includes data of potential subscribers in bank
marketing [46]. Creditcard is a credit-card fraud dataset. Donor is a valuable
projects selection dataset. Details of these two datasets can be found in
[42].

TABLE I: Statistical information of datasets. $\rm{N_{origin}}$ is the
original size. $\rm{D_{num}}$ and $\rm{D_{cat}}$ are dimensions of numerical
and categorical features. $\rm{R_{dul}}$ and $\rm{R_{miss}}$ are the rates of
duplicate and missing data. $\rm{N}$ and $\rm{D}$ are the size and dimension
after preprocessing. $\rm{N_{noise}}$ is the number of injected noise.
$\rm{R}_{anomaly}$ is the rate of anomalies.

| Before Preprocessing | After Preprocessing
---|---|---
Data | $\rm{N_{origin}}$ | $\rm{D_{num}}$ | $\rm{D_{cat}}$ |  $\rm{R_{dul}}$(%) |  $\rm{R_{miss}}$(%) | $\rm{N}$ | $\rm{D}$ | $\rm{N_{noise}}$ |  $\rm{R}_{anomaly}$(%)
Attack | 257673 | 39 | 3 | 0.00 | 0.00 | 75965 | 191 | 342 | 25.24
Bcsc | 462563 | 14 | 0 | 17.87 | 0.00 | 382122 | 14 | 2242 | 2.62
Creditcard | 284807 | 30 | 0 | 0.38 | 0.00 | 285441 | 30 | 1716 | 0.17
Diabetic | 101766 | 12 | 31 | 0.00 | 3.65 | 98575 | 115 | 522 | 12.10
Donor | 664098 | 6 | 16 | 0.04 | 12.02 | 587383 | 79 | 3317 | 6.28
Intrusion | 805050 | 38 | 3 | 73.22 | 0.00 | 216420 | 119 | 820 | 37.23
Market | 45211 | 7 | 9 | 0.00 | 0.00 | 45451 | 51 | 241 | 11.70

Data Preprocessing. We first preprocess the datasets via removing duplicated
samples or samples with missing features. For example, 73.22% of duplicates
are found on intrusion, and 12.02% of missing data are found on donor. Next,
we encode the categorical features with one-hot encoding and standardize each
numerical feature. In addition, following [42] we inject noises into the
training data to improve the robustness. Specifically, we randomly select 1%
of normal data and swap 5% features of these data. Finally, the training,
validation, testing sets are randomly divided into 6/2/2. Note that some
datasets have a part of features that leak anomaly labels. To keep consistency
in real applications, we drop one feature in attack and five features in donor
to avoid using this supervised-like information.

####  4.2.2 Benchmarks and Settings

We compare SG-AE with five benchmarks: iForest [24], RDA [13], DAGMM [10], RDP
[27], RSRAE [14]. We use the originally proposed model structures for iForest,
DAGMM, and RDP. As for RDA, RSRAE, and SG-AE, we use the same fully connected
layer settings in the autoencoder framework to maintain the consistency. To
adapt the autoencoder framework to various dataset dimensions ($\rm{D}$), we
select the sizes of the fully connected layers in the encoder from (20, 40,
80), for example, size ($\rm{D}$, 20) for creditcard, size ($\rm{D}$, 40, 20)
for market and size ($\rm{D}$, 80, 40, 20) for intrusion. The network sizes in
the decoder are symmetrical to the encoder. The scoring network in SG-AE uses
the fully connected network with size (20, 10). In terms of parameter
settings, we keep the original recommendations in iForest. To make the deep
learning models adaptable for each dataset and arrive at a convincing
conclusion, we conduct parameter searching to obtain optimal results, starting
from the recommended parameters. Specifically, RDA is tuned with $\lambda$ and
update step, while DAGMM is tuned with two $\lambda$ and mixture components
number. RSRAE is tuned with two $\lambda$ and intrinsic dimension. RDP is
tuned with node epoch, tree depth, and latent dimension. RDP set tree number
to 1 rather than in ensemble settings for fair comparisons. SG-AE is tuned
with two $\lambda$ and $\epsilon_{p}$. Additionally, we keep the same training
settings for all deep learning models. We set batch size as $1024$, the
training epochs as $100$, and the Adam optimizer with a learning rate of
$0.0001$. We select the well-trained models in the validation sets. Each
training task takes from dozens of minutes to a few hours on one NVIDIA RTX
2080 Ti GPU. The implementation code is publicly released at
https://github.com/urbanmobility/SGM.

####  4.2.3 Performance Evaluation

We use the Area Under Receiver Operating Characteristic Curve (AUC-ROC) and
the Area Under Precision-Recall Curve (AUC-PR) to evaluate the performance of
reference methods. Higher AUC values suggest better performance. The results
are averaged over 10 independent runs, in which the data sets are regenerated
randomly, to report stable performance. As shown in Table II, the results
reflect that our SG-AE outperforms other methods in all datasets. SG-AE
achieves more than 5% AUC-ROC/AUC-PR improvement over the best baselines on
attack, intrusion, market. The reliance on clean training data for DAGMM leads
to performance degradation in the anomaly-contained settings. RDP and RDA
utilize different strategies to filter anomalies during training, achieving
the second and third performance in most datasets. RSRAE, originally designed
for image data, achieves unsatisfactory performance, reflecting that it cannot
be well adapted to tabular data. The performance improvement of SG-AE benefits
from the score-guidance strategy, which enhances the score disparity and the
capability of the representation learner. Compared with the results in [27],
the performance difference is mainly caused by the construction of the
datasets. For example, we obtained 22 features from the data sources in donor
but [27] take only 10 features without detailed prepossessing steps. However,
we can have a convincing comparison result in creditcard, which is a standard
dataset and has been preprocessed in the original data source. On this
dataset, iForest and RDP perform similarly in our work and [27].

TABLE II: AUC-ROC and AUC-PR (mean$\pm$std) of SG-AE and the reference models.

AUC-ROC | AUC-PR
---|---
Datasets | iForest | DAGMM | RDP | RDA | RSRAE | SG-AE | iForest | DAGMM | RDP | RDA | RSEAE | SG-AE
Attack | 0.644$\pm$0.028 | 0.762$\pm$0.012 | 0.769$\pm$0.016 | 0.615$\pm$0.033 | 0.681$\pm$0.035 | 0.819$\pm$0.009 | 0.343$\pm$0.027 | 0.457$\pm$0.010 | 0.442$\pm$0.020 | 0.321$\pm$0.013 | 0.533$\pm$0.046 | 0.596$\pm$0.024
Bcsc | 0.754$\pm$0.033 | 0.672$\pm$0.023 | 0.894$\pm$0.003 | 0.872$\pm$0.004 | 0.840$\pm$0.023 | 0.912$\pm$0.008 | 0.080$\pm$0.026 | 0.053$\pm$0.011 | 0.605$\pm$0.024 | 0.613$\pm$0.003 | 0.137$\pm$0.048 | 0.816$\pm$0.004
Creditcard | 0.952$\pm$0.003 | 0.833$\pm$0.057 | 0.953$\pm$0.005 | 0.920$\pm$0.006 | 0.942$\pm$0.017 | 0.964$\pm$0.004 | 0.184$\pm$0.031 | 0.100$\pm$0.185 | 0.187$\pm$0.032 | 0.220$\pm$0.003 | 0.158$\pm$0.054 | 0.291$\pm$0.072
Diabetic | 0.546$\pm$0.008 | 0.514$\pm$0.020 | 0.541$\pm$0.041 | 0.538$\pm$0.005 | 0.477$\pm$0.004 | 0.548$\pm$0.007 | 0.136$\pm$0.003 | 0.128$\pm$0.010 | 0.135$\pm$0.017 | 0.132$\pm$0.003 | 0.117$\pm$0.001 | 0.142$\pm$0.004
Donor | 0.509$\pm$0.005 | 0.519$\pm$0.011 | 0.518$\pm$0.021 | 0.513$\pm$0.001 | 0.484$\pm$0.009 | 0.541$\pm$0.004 | 0.066$\pm$0.001 | 0.069$\pm$0.002 | 0.067$\pm$0.005 | 0.066$\pm$0.000 | 0.060$\pm$0.001 | 0.075$\pm$0.002
Intrusion | 0.815$\pm$0.012 | 0.860$\pm$0.046 | 0.863$\pm$0.022 | 0.614$\pm$0.027 | 0.741$\pm$0.022 | 0.906$\pm$0.016 | 0.619$\pm$0.017 | 0.801$\pm$0.034 | 0.666$\pm$0.027 | 0.478$\pm$0.012 | 0.747$\pm$0.042 | 0.904$\pm$0.035
Market | 0.652$\pm$0.016 | 0.640$\pm$0.013 | 0.685$\pm$0.021 | 0.674$\pm$0.006 | 0.536$\pm$0.013 | 0.750$\pm$0.025 | 0.189$\pm$0.011 | 0.211$\pm$0.007 | 0.227$\pm$0.022 | 0.210$\pm$0.006 | 0.129$\pm$0.005 | 0.270$\pm$0.030

Figure 5 presents the score distribution of SG-AE. We can see that the anomaly
score distributions of normal data (blue bars) and abnormal data (red bars)
are notably separated on most datasets. For numerical comparison between the
typical methods, we use KS test to reflect the distribution distance between
normal and abnormal scores. KS test is a statistical hypothesis test that
checks whether two samples differ in distributions [47]. A higher KS index
indicates a larger score difference. As shown in Table III, the average values
of the KS of SG-AE on the seven datasets are $0.539$, surpassing the $0.355$
of AE, $0.372$ of AE, $0.336$ of RDA, and $0.415$ of RDP. This suggests that
our scoring network successfully enhanced the disparity between the normal and
abnormal data. A larger KS index correlates with better performance showed in
Table II, especially on the bcsc, creditcard, intrusion datasets. Taken
together, these results demonstrate the effectiveness of the score guidance
design.

Figure 5: The anomaly score distribution of actual normal and abnormal data on
the seven datasets. As expected, the SG-AE learns relative large anomaly
scores for the abnormal data (red), but small scores for the normal data
(blue). TABLE III: KS index between the anomaly score distributions of the
normal and abnormal data in various models.

KS index | Attack | Bcsc | Creditcard | Diabetic | Donor | Intrusion | Market
---|---|---|---|---|---|---|---
iForest | 0.207 | 0.456 | 0.805 | 0.076 | 0.035 | 0.717 | 0.188
AE | 0.224 | 0.089 | 0.857 | 0.082 | 0.086 | 0.843 | 0.422
RDA | 0.211 | 0.790 | 0.782 | 0.064 | 0.024 | 0.214 | 0.265
RDP | 0.346 | 0.741 | 0.870 | 0.103 | 0.056 | 0.450 | 0.340
SG-AE | 0.570 | 0.791 | 0.882 | 0.114 | 0.089 | 0.864 | 0.509

To intuitively understand the anomaly detection results, we utilize t-SNE for
dimensionality reduction and visualize the seven datasets in 2D space, as
shown in Figure 6. Data samples are colored with their true labels or anomaly
scores generated by our SG-AE and RDP, which performs best among the
baselines. The light color indicates these samples can not be easily
distinguished as normal or abnormal data. One can select different thresholds
to identify the abnormal samples, according to the datasets and needs. In
Figure 6, we observe that t-SNE can well segregate the anomalies in the 2D
space on half of the datasets, and SG-AE presents more similar segregation to
the ground truth than RDP, especially on bcsc and intrusion. These results are
consistent with the results in Table II, reflecting the anomaly detection
capabilities of SG-AE. We also notice that the anomalies are not segregated
well on diabetic and donor, indicating the challenge of anomaly detection.

Figure 6: The t-SNE visualization analysis. The three rows refer to the ground
truth (GT), results of RDP and SG-AE, respectively. Red represents abnormal,
while blue represents normal. The darker color indicates the larger
probability to be normal or abnormal.

####  4.2.4 Comparison with Variants

In SG-AE, we introduce the score-guidance strategy to anomaly scores with the
reconstruction error as a self-supervised signal. To examine the effect of the
learnable anomaly score, we remove the scoring network and use the
reconstruction error as the anomaly score, keeping the regularizer in Eq.
(11). That means regularizer is used to guide the reconstruction error. This
variant of SG-AE is named as SG-AErecon. We also test AE, the original
autoencoder without the score-guided loss and the scoring network.

We next devise the other two variants, SG-AEnormal and SG-AElognormal, to
examine the score distribution assumption. The anomaly scores appear to have a
normal or lognormal distribution in the statistical histogram displayed in
Figure 5. Inspired by this, we assume the anomaly scores follow a normal or
lognormal distribution in our SG-AE. The outputs of the scoring network are
altered to the mean and standard deviation of scores, denoted by $\mu$ and
$\sigma$. We utilize KL divergence to measure the distribution difference,
thus the score guided regularization (10) can be written as follows.

$L_{\rm{SE}}(\bm{x},\bm{\tilde{x}},\mu,\sigma)=\begin{cases}\rm{KL}\big{(}D(\mu,\sigma),D(0,1)\big{)},&\big{\|}\bm{x}-\bm{\tilde{x}}\big{\|}_{2}<\epsilon\\\
\lambda_{a}\cdot\max(0,a-\mu),&\big{\|}\bm{x}-\bm{\tilde{x}}\big{\|}_{2}\geq\epsilon\end{cases}.$

(15)

Table IV compares the performance of the original SG-AE and its variants. The
original SG-AE performs best on five datasets. SG-AEnormal and SG-AElognormal
have similar performances, which are slightly worse than SG-AEoriginal in most
cases. SG-AErecon achieves the best performances on diabetic and donor.
However, the performance of SG-AErecon has large variances across datasets,
which is caused by the conflict of these two loss functions. The
reconstruction loss tends to reduce the reconstruction error, but the score
guidance loss attempt to enlarge the reconstruction error for anomalies. This
conflict brings challenges to the update of network states. The results of SG-
AEoriginal, SG-AEnormal, SG-AElognormal are significantly better than AE,
which demonstrates the effectiveness of the scoring network.

TABLE IV: AUC-ROC and AUC-PR (mean±std) of SG-AE and its variants.

AUC-ROC | AUC-PR
---|---
Datasets | SG-AEoriginal | SG-AEnormal | SG-AElognormal | SG-AErecon | AE | SG-AEoriginal | SG-AEnormal | SG-AElognormal | SG-AErecon | AE
Attack | 0.819$\pm$0.009 | 0.772$\pm$0.06 | 0.745$\pm$0.09 | 0.639$\pm$0.112 | 0.561$\pm$0.037 | 0.596$\pm$0.024 | 0.436$\pm$0.0724 | 0.412$\pm$0.079 | 0.436$\pm$0.106 | 0.303$\pm$0.016
Bcsc | 0.912$\pm$0.008 | 0.897$\pm$0.006 | 0.898$\pm$0.006 | 0.625$\pm$0.288 | 0.878$\pm$0.010 | 0.816$\pm$0.004 | 0.781$\pm$0.050 | 0.751$\pm$0.078 | 0.229$\pm$0.281 | 0.617$\pm$0.004
Creditcard | 0.964$\pm$0.004 | 0.940$\pm$0.017 | 0.928$\pm$0.027 | 0.716$\pm$0.158 | 0.903$\pm$0.013 | 0.291$\pm$0.072 | 0.137$\pm$0.060 | 0.128$\pm$0.046 | 0.084$\pm$0.107 | 0.210$\pm$0.004
Diabetic | 0.548$\pm$0.007 | 0.540$\pm$0.015 | 0.541$\pm$0.013 | 0.559$\pm$0.077 | 0.535$\pm$0.007 | 0.142$\pm$0.004 | 0.133$\pm$0.009 | 0.138$\pm$0.005 | 0.156$\pm$0.043 | 0.131$\pm$0.003
Donor | 0.541$\pm$0.004 | 0.532$\pm$0.006 | 0.536$\pm$0.006 | 0.559$\pm$0.063 | 0.486$\pm$0.045 | 0.075$\pm$0.002 | 0.072$\pm$0.002 | 0.073$\pm$0.002 | 0.076$\pm$0.012 | 0.063$\pm$0.007
Intrusion | 0.906$\pm$0.016 | 0.712$\pm$0.077 | 0.695$\pm$0.037 | 0.748$\pm$0.312 | 0.568$\pm$0.036 | 0.904$\pm$0.035 | 0.546$\pm$0.055 | 0.541$\pm$0.024 | 0.730$\pm$0.242 | 0.461$\pm$0.017
Market | 0.750$\pm$0.025 | 0.647$\pm 0.018$ | 0.644$\pm$0.013 | 0.600$\pm$0.076 | 0.677$\pm$0.008 | 0.270$\pm$0.030 | 0.191$\pm$0.009 | 0.187$\pm$0.006 | 0.191$\pm$0.065 | 0.215$\pm$0.007

####  4.2.5 Sensitivity of Parameters

In this part, we examine the sensitivity of SG-AE to different parameters. For
convenience, we divide the four parameters into two groups, $\epsilon_{p}$ and
$a$ in group one, $\rm\lambda_{SE}$ and $\lambda_{a}$ in group two. We first
test the group one by fixing $\rm\lambda_{SE}=0.01$ and $\lambda_{a}=18$, then
test the group two with fixed $\epsilon_{p}$ and $a$.

Figure 7: The AUC-ROC values for SG-AE with various sets of parameters on
three datasets, attack, credictcard, and martket. We first fix
$\lambda_{SE}=0.01$ and $\lambda_{a}=18$ for each dataset and train the SG-AE
with different values of $\epsilon_{p}$ and $a$. Then we fix $\epsilon_{p}$
and $a$, and train the SG-AE with different values of $\lambda_{SE}$ and
$\lambda_{a}$.

Figure 7 reports the AUC-ROC values of the two-step testing on three datasets,
attack, creditcard, and market. The darker colors suggest better performance.
We note that the state-of-the-art results can be achieved in two rounds of
parameter searching, which indicates that the parameter searching is not very
complicated or time-consuming. In addition, we find that (i) $\epsilon_{p}$
and $\lambda_{a}$ have relatively greater impact; (ii) $\epsilon_{p}$ can be
smaller than the normal data ratio, especially with a little prior knowledge
of the dataset; (iii) in consistent with the the analysis in methodology, the
performance of SG-AE changes lightly when $a>5$; (iv) the sensitivity also
behaves differently for these datasets owning to the complexity and
characteristics.

####  4.2.6 Robustness to Different Anomaly Rates

As we intend to learn the disparity between the normal and abnormal data using
a scoring network, we can expect that the performance of SG-AE would be robust
to the ratio of the abnormal data. To validate this expectation, we reorganize
the two datasets with high anomaly rates, attack and intrusion, and compare
SG-AE with iForest, AE, RDP on these datasets with different anomaly rates.
Specifically, we randomly drop anomaly samples in the training set, adjusting
the anomaly rate to $[5\%,10\%,15\%,20\%,25\%]$ on attack and
$[5\%,10\%,15\%,20\%,25\%,30\%,35\%]$ on intrusion. The validation and testing
sets keep the original anomaly rate.

Figure 8 presents the comparison results in different anomaly rates.
Obviously, SG-AE achieves a more stable performance than other competing
models, while AE has the fast AUC-ROC performance degradation on both
datasets. The strong sensitivity of AE to the anomaly rate mainly due to its
intention to reconstruct all input data, while a large fraction of abnormal
data hinders the training of the reconstruction model. We also observe that
the AUC-ROC values of iForest and RDP drop faster in attack than in intrusion,
suggesting that the robustness to anomaly rates also depends on the
characteristic of data.

Figure 8: The AUC-ROC values of selected models in different anomaly rates on
two datasets attack and intrusion. SG-AE achieves stable AUC-ROC with various
anomaly rates.

###  4.3 Extension to SOTA models

To examine the transferability of the scoring network to other models, we
evaluate the performances of three extended models in the seven tabular
datasets.

Settings. We extend the scoring network to three state-of-the-art models:
DAGMM, RDA, and RSRAE. The extended models are named score-guided DAGMM (SG-
DAGMM), score-guided RDA (SG-RDA), and score-guided RSRAE (SG-RSRAE). For the
input of the scoring network, SG-DAGMM takes the same input as the compression
network in DAGMM, SG-RDA takes the output of the encoder, and SG-RSRAE takes
the output of the RSR layer. Here the datasets are configured same as the
experiments of SG-AE. We also search parameters to adapt the SG-Models to
these datasets. Specifically, SG-RDA is tuned with its original $\lambda$ and
$\lambda_{\rm{SE}}$, SG-DAGMM and SG-RSRAE are tuned with their two original
$\lambda$ and $\lambda_{\rm{SE}}$.

Performance evaluation. The results of AUC-ROC and AUC-PR are shown in table
V. Compared with the original model, these SG-Models achieve varying degrees
of improvement in most datasets. These results suggest that the score-guidance
strategy can better handle the contaminated data. In addition, SG-DAGMM
obtains more obvious performance improvements and surpasses SG-AE on some
datasets. This is mainly because the scoring network enables DAGMM to deal
with the anomaly contamination and generates better data representation for
the Gaussian Mixture Model. These experimental results not only demonstrate
the effectiveness of the scoring network design, but also reflect that the
scoring network can be incorporated into different unsupervised anomaly
detection models.

TABLE V: AUC-ROC and AUC-PR (mean$\pm$std) of SG-SOTA models.

AUC-ROC | AUC-PR
---|---
Datasets | SG-AE | SG-RDA | SG-RSRAE | SG-DAGMM | SG-AE | SG-RDA | SG-RSRAE | SG-DAGMM
Attack | 0.819$\pm$0.009 | 0.816$\pm$0.016 (32.7%) | 0.771$\pm$0.064 (13.2%) | 0.783$\pm$0.044 (44.4%) | 0.596$\pm$0.024 | 0.534$\pm$0.027 (66.4%) | 0.529$\pm$0.096 (-0.8%) |  0.660$\pm$0.035 (2.8%)
Bcsc | 0.912$\pm$0.008 | 0.905$\pm$0.004 (3.8%) | 0.868$\pm$0.043 (3.3%) | 0.785$\pm$0.117 (16.8%) | 0.816$\pm$0.004 | 0.805$\pm$0.001 (31.3%) | 0.510$\pm$0.219 (272.3%) | 0.421$\pm$0.380 (694.3%)
Creditcard | 0.964$\pm$0.004 | 0.940$\pm$0.010 (2.2%) | 0.930$\pm$0.019 (-1.3%) | 0.915$\pm$0.021 (9.8%) | 0.291$\pm$0.072 | 0.202$\pm$0.083 (-7.3%) | 0.222$\pm$0.100 (40.5%) |  0.494$\pm$0.169 (394.0%)
Diabetic | 0.548$\pm$0.007 | 0.561$\pm$0.027 (4.3%) | 0.545$\pm$0.012 (14.3%) |  0.581$\pm$0.066 (13.0%) | 0.142$\pm$0.004 | 0.143$\pm$0.013 (8.3%) | 0.135$\pm$0.005 (15.4%) |  0.168$\pm$0.049 (31.3%)
Donor | 0.541$\pm$0.004 | 0.540$\pm$0.007 (5.3%) |  0.597$\pm$0.059 (23.3%) | 0.568$\pm$0.049 (9.4%) | 0.075$\pm$0.002 | 0.078$\pm$0.003 (18.2%) |  0.081$\pm$0.011 (35.0%) | 0.076$\pm$0.005 (10.1%)
Intrusion | 0.906$\pm$0.016 | 0.899$\pm$0.019 (46.4%) | 0.808$\pm$0.144 (9.0%) |  0.976$\pm$0.003 (13.5%) | 0.904$\pm$0.035 | 0.758$\pm$0.030 (58.6%) | 0.736$\pm$0.188 (-1.5%) |  0.971$\pm$0.003 (21.2%)
Market | 0.750$\pm$0.025 | 0.747$\pm$0.033 (10.8%) | 0.733$\pm$0.030 (36.8%) | 0.698$\pm$0.003 (9.1%) | 0.270$\pm$0.030 | 0.251$\pm$0.031 (19.5%) | 0.238$\pm$0.033 (84.5%) | 0.203$\pm$0.011 (-3.8%)

###  4.4 Extension to document and image tasks

To examine the transferability to different tasks, we compare SG-AE with
iForest, DAGMM, RDA, RSRAE, and RDP on document and image datasets. Settings.
For the document task, we utilize datasets, news and reuters [14]. News
involves newsgroup documents with 20 different labels and reuters contains 5
text categories. Following the preprocessing steps in [14], news and reuters
are randomly split into 360 documents per class and the documents are embedded
into vectors of size $10,000$ and $26,147$, respectively. In each round of
testing, we choose one class as normal data in turns and randomly select
abnormal samples from other classes. By collecting different numbers of
abnormal data, we compare the SG-AE with baselines in the document datasets
with varying anomaly rates, $[5\%,10\%,15\%,20\%,25\%]$. For image task, we
take mnist dataset as an example [13]. mnist consists of 5124 instances, with
the anomaly rate of $5.2\%$. The normal data are the image of digit ”4” and
the abnormal data are other digits. For each task, we average the AUC-ROC and
AUC-PR values over 10 independent runs. The training and testing sets are 8/2.
The layer sizes of the autoencoder framework are [1024, 256, 64, 20] in
document tasks and [128, 64, 32] in the image task. The batch size is set as
32 and other settings are consistent with the previous experiments. We also
conduct parameter searching to achieve nearly optimal results.

Figure 9: AUC-ROC and AUC-PR values on document data.

Performance evaluation. Figure 9 illustrates the results for document
datasets. SG-AE achieves the best performance in reuters. In 20news, SG-AE
performs slightly weaker than RSRAE and RDA at a smaller anomaly rate but
performs better at a larger anomaly rate. This might be due to the different
granularity when handling anomalies. Specifically, RSRAE and RDA filter
abnormal parts for each sample at the feature level, which is effective for
datasets with large feature correlation (such as document or image datasets).
SG-AE simply guides the sample distribution at the data level. In fact, the
feature-level and the data-level strategies are not in conflict. The
combination of these two strategies can be further studied in the future. The
results for the image dataset are shown in Table VI. Similar to the document
datasets, SG-AE has competitive AUC-ROC, but performs weaker than RSRAE in
term of AUC-PR. Therefore, we further evaluate SG-RSRAE, achieving performance
improvements of $1.3\%$ on AUC-ROC and $19.7\%$ on AUC-PR over RSRAE. These
results confirm the effectiveness of the score-guidance strategy in different
tasks.

TABLE VI: AUC-ROC and AUC-PR values on mnist dataset. Metrics | AUC-ROC | AUC-PR
---|---|---
iForest | 0.873$\pm$0.014 | 0.372$\pm$0.036
DAGMM | 0.758$\pm$0.040 | 0.235$\pm$0.036
RDP | 0.888$\pm$0.024 | 0.475$\pm$0.065
RDA | 0.912$\pm$0.010 | 0.556$\pm$0.038
RSRAE | 0.939$\pm$0.007 | 0.615$\pm$0.026
SG-AE | 0.939$\pm$0.005 | 0.563$\pm$0.015
SG-RSRAE | 0.951$\pm$0.010 | 0.736$\pm$0.062

##  5 Conclusion and future work

Targeting the unsupervised anomaly detection task, this work devised an
effective scoring network with score-guided regularization. The scoring
network is able to learn the disparity of normal and abnormal data and the
disparity is gradually enhanced during the network training. The scoring
network can be integrated into different unsupervised anomaly detection
methods. We next proposed a representative instantiation of incorporating the
scoring network into an autoencoder framework, namely score-guided autoencoder
(SG-AE). We first conducted experiments on synthetic datasets to examine the
effectiveness of the score-guidance strategy. The results show that the
anomaly scores disparity between normal and abnormal data continues to expand
during training, especially in the transition field. Then comprehensive
experiments are conducted on seven tabular datasets, suggesting that the
proposed SG-AE is competitive with the state-of-the-art methods in terms of
both AUC values and the Kolmogorov–Smirnov statistical index. We also analyzed
the sensitivity of the introduced parameters in our scoring network and found
that different datasets can share similar configurations. In addition, as the
scoring network can learn the disparity between normal and abnormal data, we
expect that SG-AE can work well even with a large fraction of anomaly data. To
validate this expectation, we then tested the SG-AE on two datasets with
varying anomaly rates, indicating that SG-AE is more robust to the anomaly
rate than the other three baselines. In order to present the transferability
of the scoring network on different unsupervised methods, we then applied the
scoring network to three state-of-the-art methods and the experimental results
approve the performance improvement. Moreover, we applied the scoring network
to two document datasets and one image dataset to present the transferability
to different anomaly detection tasks. Experimental results show that SG-Models
is comparable to the state-of-the-art methods, especially for datasets with
large anomaly rates.

There are several improvements and potential extensions that merit further
study: (1) Adjusting the $\lambda$ parameters to balance the effects of
several loss function terms is challenging when applying the scoring network
to a method with multiple loss functions. Potential solution could be
utilizing the multi-objective optimization techniques [48, 49], which attempt
to balance the trade-offs between several objectives. Hyperparameter tuning
techniques [50, 51], which map hyperparameters into loss functions and
optimize them during training, are also worthy of consideration to deal with
the problem of too many hyperparameters. (2) The score-guidance strategy
deserves to be further applied to convolutional frameworks or sequential
frameworks to evaluate the performance changes in various anomaly detection
tasks with image, time-series, or graph datasets.

## Acknowledgments

This work was jointly supported by the Shanghai Municipal Science and
Technology Major Project (2021SHZDZX0102), the Science and Technology
Commission of Shanghai Municipality Project (2051102600), and the National Key
Research and Development Program of China (2020YFC2008701).

## References

  * [1] R. Chalapathy and S. Chawla, “Deep learning for anomaly detection: A survey,” _arXiv preprint arXiv:1901.03407_ , 2019.
  * [2] G. Pang, C. Shen, L. Cao, and A. v. d. Hengel, “Deep learning for anomaly detection: A review,” _arXiv preprint arXiv:2007.02500_ , 2020.
  * [3] L. Li, J. Yan, H. Wang, and Y. Jin, “Anomaly detection of time series with smoothness-inducing sequential variational auto-encoder,” _IEEE transactions on neural networks and learning systems_ , 2020.
  * [4] W.-K. Wong, A. Moore, G. Cooper, and M. Wagner, “Rule-based anomaly pattern detection for detecting disease outbreaks,” in _AAAI/IAAI_ , 2002, pp. 217–223.
  * [5] F. Falcão, T. Zoppi, C. B. V. Silva, A. Santos, B. Fonseca, A. Ceccarelli, and A. Bondavalli, “Quantitative comparison of unsupervised anomaly detection algorithms for intrusion detection,” in _Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing_ , 2019, pp. 318–327.
  * [6] M. Ahmed, N. Choudhury, and S. Uddin, “Anomaly detection on big data in financial markets,” in _2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)_. IEEE, 2017, pp. 998–1001.
  * [7] L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller, “A unifying review of deep and shallow anomaly detection,” _arXiv preprint arXiv:2009.11732_ , 2020.
  * [8] I. Golan and R. El-Yaniv, “Deep anomaly detection using geometric transformations,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , 2018, pp. 9781–9791.
  * [9] L. Ruff, R. Vandermeulen, N. Goernitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft, “Deep one-class classification,” in _International conference on machine learning_. PMLR, 2018, pp. 4393–4402.
  * [10] B. Zong, Q. Song, M. R. Min, W. Cheng, C. Lumezanu, D. Cho, and H. Chen, “Deep autoencoding gaussian mixture model for unsupervised anomaly detection,” in _ICLR_ , 2018.
  * [11] S. Goyal, A. Raghunathan, M. Jain, H. V. Simhadri, and P. Jain, “Drocc: Deep robust one-class classification,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 3711–3721.
  * [12] L. Bergman and Y. Hoshen, “Classification-based anomaly detection for general data,” _arXiv preprint arXiv:2005.02359_ , 2020.
  * [13] C. Zhou and R. C. Paffenroth, “Anomaly detection with robust deep autoencoders,” in _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2017, pp. 665–674.
  * [14] C.-H. Lai, D. Zou, and G. Lerman, “Robust subspace recovery layer for unsupervised anomaly detection,” in _ICLR_ , 2020.
  * [15] E. Eskin, “Anomaly detection over noisy data using learned probability distributions,” in _In Proceedings of the International Conference on Machine Learning_. Citeseer, 2000.
  * [16] J. Kim and C. D. Scott, “Robust kernel density estimation,” _The Journal of Machine Learning Research_ , vol. 13, no. 1, pp. 2529–2565, 2012.
  * [17] S. Zhai, Y. Cheng, W. Lu, and Z. Zhang, “Deep structured energy based models for anomaly detection,” in _The 33rd International Conference on Machine Learning_ , 2016, pp. 1100–1109.
  * [18] C. C. Aggarwal, “Outlier analysis,” in _Data mining_. Springer, 2015, pp. 237–263.
  * [19] D. Hendrycks, M. Mazeika, and T. Dietterich, “Deep anomaly detection with outlier exposure,” in _ICLR_ , 2018.
  * [20] S. Ramaswamy, R. Rastogi, and K. Shim, “Efficient algorithms for mining outliers from large data sets,” in _ACM International Conference on Management of Data_ , 2000, pp. 427–438.
  * [21] X. Gu, L. Akoglu, and A. Rinaldo, “Statistical analysis of nearest neighbor methods for anomaly detection,” _arXiv preprint arXiv:1907.03813_ , 2019\.
  * [22] M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “Lof: identifying density-based local outliers,” in _ACM International Conference on Management of Data_ , 2000, pp. 93–104.
  * [23] L. Bergman, N. Cohen, and Y. Hoshen, “Deep nearest neighbor anomaly detection,” _arXiv preprint arXiv:2002.10445_ , 2020.
  * [24] F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation forest,” in _2008 eighth ieee international conference on data mining_. IEEE, 2008, pp. 413–422.
  * [25] S. Guha, N. Mishra, G. Roy, and O. Schrijvers, “Robust random cut forest based anomaly detection on streams,” in _International conference on machine learning_. PMLR, 2016, pp. 2712–2721.
  * [26] G. Pang, L. Cao, L. Chen, and H. Liu, “Learning representations of ultrahigh-dimensional data for random distance-based outlier detection,” in _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, 2018, pp. 2041–2050.
  * [27] H. Wang, G. Pang, C. Shen, and C. Ma, “Unsupervised representation learning by predicting random distances,” in _International Joint Conferences on Artificial Intelligence_ , 2020.
  * [28] P. Baldi, “Autoencoders, unsupervised learning, and deep architectures,” in _Proceedings of ICML workshop on unsupervised and transfer learning_ , 2012, pp. 37–49.
  * [29] K. Tian, S. Zhou, J. Fan, and J. Guan, “Learning competitive and discriminative reconstructions for anomaly detection,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, 2019, pp. 5167–5174.
  * [30] J. An and S. Cho, “Variational autoencoder based anomaly detection using reconstruction probability,” _Special Lecture on IE_ , vol. 2, no. 1, pp. 1–18, 2015.
  * [31] H. Xu, W. Chen, N. Zhao, Z. Li, J. Bu, Z. Li, Y. Liu, Y. Zhao, D. Pei, Y. Feng _et al._ , “Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications,” in _Proceedings of the 2018 World Wide Web Conference_ , 2018, pp. 187–196.
  * [32] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs, “Unsupervised anomaly detection with generative adversarial networks to guide marker discovery,” in _International conference on information processing in medical imaging_. Springer, 2017, pp. 146–157.
  * [33] M. Sabokrou, M. Khalooei, M. Fathy, and E. Adeli, “Adversarially learned one-class classifier for novelty detection,” in _CVPR_ , 2018, pp. 3379–3388.
  * [34] D. Li, D. Chen, J. Goh, and S.-k. Ng, “Anomaly detection with generative adversarial networks for multivariate time series,” _arXiv preprint arXiv:1809.04758_ , 2018.
  * [35] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _The journal of machine learning research_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [36] Y. Wen, S. Li, and K. Jia, “Towards understanding the regularization of adversarial robustness on neural networks,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 10 225–10 235.
  * [37] D. Arpit, Y. Zhou, H. Ngo, and V. Govindaraju, “Why regularized auto-encoders learn sparse representation?” in _International Conference on Machine Learning_. PMLR, 2016, pp. 136–144.
  * [38] W. Yu, C. Zheng, W. Cheng, C. C. Aggarwal, D. Song, B. Zong, H. Chen, and W. Wang, “Learning deep network representations with adversarially regularized autoencoders,” in _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018, pp. 2663–2671.
  * [39] S. Qian, G. Li, W.-M. Cao, C. Liu, S. Wu, and H.-S. Wong, “Improving representation learning in autoencoders via multidimensional interpolation and dual regularizations.” in _IJCAI_ , 2019, pp. 3268–3274.
  * [40] H. Xu, D. Luo, R. Henao, S. Shah, and L. Carin, “Learning autoencoders with relational regularization,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 10 576–10 586.
  * [41] R. Chalapathy, A. K. Menon, and S. Chawla, “Robust, deep and inductive anomaly detection,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2017, pp. 36–51.
  * [42] G. Pang, C. Shen, and A. van den Hengel, “Deep anomaly detection with deviation networks,” in _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, 2019, pp. 353–362.
  * [43] W. E. Barlow, E. White, R. Ballard-Barbash, P. M. Vacek, L. Titus-Ernstoff, P. A. Carney, J. A. Tice, D. S. Buist, B. M. Geller, R. Rosenberg _et al._ , “Prospective breast cancer risk prediction model for women undergoing screening mammography,” _Journal of the National Cancer Institute_ , vol. 98, no. 17, pp. 1204–1214, 2006.
  * [44] B. Strack, J. P. DeShazo, C. Gennings, J. L. Olmo, S. Ventura, K. J. Cios, and J. N. Clore, “Impact of hba1c measurement on hospital readmission rates: analysis of 70,000 clinical database patient records,” _BioMed research international_ , 2014.
  * [45] D. Dua and C. Graff, “UCI machine learning repository,” 2017. [Online]. Available: http://archive.ics.uci.edu/ml
  * [46] N. Moustafa and J. Slay, “Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set),” in _2015 military communications and information systems conference (MilCIS)_. IEEE, 2015, pp. 1–6.
  * [47] D. M. dos Reis, P. Flach, S. Matwin, and G. Batista, “Fast unsupervised online drift detection using incremental kolmogorov-smirnov test,” in _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2016, pp. 1545–1554.
  * [48] M. Konakovic Lukovic, Y. Tian, and W. Matusik, “Diversity-guided multi-objective bayesian optimization with batch evaluations,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [49] L. Liu, Y. Li, Z. Kuang, J.-H. Xue, Y. Chen, W. Yang, Q. Liao, and W. Zhang, “Towards impartial multi-task learning,” in _International Conference on Learning Representations_ , 2021.
  * [50] I. Shavitt and E. Segal, “Regularization learning networks: deep learning for tabular datasets,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 1379–1389.
  * [51] M. Mackay, P. Vicol, J. Lorraine, D. Duvenaud, and R. Grosse, “Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions,” in _International Conference on Learning Representations_ , 2019.

|  Zongyuan Huang, received the B.E. degree from Huazhong University of
Science and Technology in 2019, is currently working toward the Ph.D. degree
in the Department of Electronic Engineering, Shanghai Jiao Tong University,
China. His research interests lie in machine learning and data mining with
particular focuses on anomaly detection, urban computing, mobility data
mining.
---|---
|  Baohua Zhang received her Doctor’s Degree in Computer Software and Theory
from Fudan University in 2009. She joined the Software Development Center of
ICBC in 2009 and is currently working in the Big Data and AI Lab of ICBC as a
technical manager. She is now the team leader of big data and AI technology
scouting in the financial field.
---|---
|  Guoqiang Hu received his M.Sc. and Dr.-Ing. Degrees in Electrical
Engineering from the University of Stuttgart, Germany in 2002 and 2007,
respectively. From 2008 to 2010, he was a research fellow in the Norwegian
University of Science and Technology (NTNU), focusing on the domain of network
architecture design and quality-of-service provisioning. From 2010 to 2021, he
was a research staff member in IBM China Research Lab and led innovation
projects in the field of Internet-of-Things and computer vision. He joined the
Big Data and AI Lab of ICBC in 2021 and is the technical director of
technology scouting and computer vision.
---|---

|  Longyuan Li received the B.E. degree in Electronic Engineering from
Huazhong University of Science and Technology in 2015. He is currently working
toward the Ph.D. degree in the Department of Electronic Engineering, Shanghai
Jiao Tong University, China. His research interests include machine learning
and data mining. He focuses on probabilistic models and Bayesian non-
parametric models, for sequential data such as multi-dimensional time-series.
He is also interested in deep probabilistic graphical models and uncertainty
estimation, with first-authored publications in IJCAI’19, AAAI’21, IEEE TNNLS.
---|---
|  Yanyan Xu, received the Ph.D. degree from the Department of Automation,
Shanghai Jiao Tong University in 2015. He is now an associate professor in the
AI Institute, Shanghai Jiao Tong University. Prior to joining SJTU, he was a
postdoctoral associate in the Human Mobility and Networks (HuMNet) Lab in the
Department of Urban and Regional Planning at UC Berkeley and the Department of
Civil and Environmental Engineering at MIT from 2015 to 2020. He was also a
guest postdoctoral fellow in the Energy Analysis and Environmental Impacts
Division, Lawrence Berkeley National Laboratory from 2017 to 2018. His
research lies in human mobility and urban computing, with particular emphasis
placed on the use of massive trajectory data in Urban Science, Transportation,
Energy, and Environment, from interdisciplinary perspectives. His work has
been published in Nature Energy, Science Advances, J. Roy. Soc. Interface,
IEEE Trans. ITS, TRC, and IJCAI, among others.
---|---
|  Yaohui Jin was once a Technical Staff Member with Bell Labs Research China.
After that he joined Shanghai Jiao Tong University in 2002, where he is a
Professor with the State Key Laboratory of Advanced Optical Communication
Systems and Networks and the Deputy Director of Network and Information
Center. His research interests include civic engagement and open innovation,
cloud computing network architecture, and streaming data analysis. He is the
Founder of OMNILab, which is an open innovation lab focusing on data analysis.
He has published over 100 technical papers in leading conferences and journals
and is the owner of over 10 patents. In 2014, OMNILab won the champion of CCF
national big data challenge among nearly 1000 teams, and was the champion of
the Shanghai open data innovation and creation competition. He has served over
10 technical committees. He enthuses public service and science
popularization, actively promotes crowd engaged innovation, and
interdisciplinary collaboration.
---|---