# Zero-Shot Certified Defense against Adversarial Patches with Vision
Transformers

Yuheng Huang
University of Alberta
Edmonton, Canada
<EMAIL_ADDRESS>&Yuanchun Li
Institute for AI Industry Research (AIR)
Tsinghua University, Beijing, China
<EMAIL_ADDRESS>

###### Abstract

Adversarial patch attack aims to fool a machine learning model by arbitrarily
modifying pixels within a restricted region of an input image. Such attacks
are a major threat to models deployed in the physical world, as they can be
easily realized by presenting a customized object in the camera view.
Defending against such attacks is challenging due to the arbitrariness of
patches, and existing provable defenses suffer from poor certified accuracy.
In this paper, we propose PatchVeto, a zero-shot certified defense against
adversarial patches based on Vision Transformer (ViT) models. Rather than
training a robust model to resist adversarial patches which may inevitably
sacrifice accuracy, PatchVeto reuses a pretrained ViT model without any
additional training, which can achieve high accuracy on clean inputs while
detecting adversarial patched inputs by simply manipulating the attention map
of ViT. Specifically, each input is tested by voting over multiple inferences
with different attention masks, where at least one inference is guaranteed to
exclude the adversarial patch. The prediction is certifiably robust if all
masked inferences reach consensus, which ensures that any adversarial patch
would be detected with no false negative. Extensive experiments have shown
that PatchVeto is able to achieve high certified accuracy (e.g. 67.1% on
ImageNet for 2%-pixel adversarial patches), significantly outperforming state-
of-the-art methods. The clean accuracy is the same as vanilla ViT models
(81.8% on ImageNet) since the model parameters are directly reused. Meanwhile,
our method can flexibly handle different adversarial patch sizes by simply
changing the masking strategy.

##  1 Introduction

Deep neural networks (DNNs) are increasingly deployed in the physical world
for safety-critical computer vision tasks. Examples include face
authentication on smartphones (Sun et al., 2016), driving assistance on
autonomous cars (Wei et al., 2019), and intruder detection on surveillance
cameras (Chung et al., 2017). Unfortunately, DNNs are widely known to be
vulnerable against evasion attacks (Szegedy et al., 2014; Papernot et al.,
2018), where an adversary can easily induce misclassifications by adding a
small perturbation to the input image. How to defend against such attacks has
been a challenging research problem due to the poor interpretability of DNNs.

Among all types of evasion attacks, adversarial patch attacks (Brown et al.,
2017; Karmon et al., 2018) have attracted much attention due to their
practicality in the physical world. Unlike classic adversarial attacks that
seek to generate a global $L_{p}$-bounded perturbation ($p=1,2,\infty,...$),
an adversarial patch can arbitrarily modify the pixels within a restricted
region of the input image. For example, an attacker can fool the DNN in a
self-driving car by attaching a printed sticker onto the road signs (Eykholt
et al., 2018), or bypass face authentication by presenting a customized object
in front of the camera (Komkov & Petiushko, 2021). Various heuristic defenses
(Naseer et al., 2019; Rao et al., 2020) have been designed for adversarial
patch attacks. However, these heuristic approaches are unable to guarantee
robustness against unseen adaptive attacks (Chiang et al., 2020).

There are several provable defenses proposed for adversarial patches. Most of
them (Zhang et al., 2020; Levine & Feizi, 2020; Xiang et al., 2021; Metzen &
Yatsura, 2021) aim to design a model that can guarantee to make consistent
predictions on images with or without an adversarial patch. These approaches
are mostly based on the insight that the prediction can be made by aggregating
local features extracted from small independent receptive fields. The
certified robustness is achieved by ensuring that the aggregated result will
not be dominated by a small number of potentially adversarial receptive
fields. As a result, they have to significantly compromise the prediction
accuracy due to the lack of global features. The most recent work (Metzen &
Yatsura, 2021) in this type can only achieve a certified accuracy of 26.0% on
ImageNet under 2%-pixel adversarial patches.

Instead of attempting to hazardously make imprecise predictions in presence of
adversarial patches, we argue that it might be more reasonable to find a
boundary of when the inference can be trusted. For example, imagine a DNN is
used for driving assistance, it would be desirable for the model to promise
reliability and precision when it gives a driving suggestion, while informing
the driver about the potential risk if the input is potentially harmful. Such
defense can be viewed as a provable detection technique, which attempts to
guarantee the perfect recall (i.e. no false negative) of adversarial input
detection. The Minority Reports Defense (McCoyd et al., 2020), PatchGuard++
(Xiang & Mittal, 2021), and ScaleCert (Han et al., 2021) are designed for such
purpose, which partially occludes the image around each candidate patch
location and analyzes the predictions of all occluded images to detect
adversarial patches. However, the Minority Reports Defense cannot scale to
complex high-resolution images (e.g. ImageNet) due to the training difficulty
and heavy computation overhead, and PatchGuard++ and ScaleCert are based on
CNN backbones with small receptive fields (e.g. BagNet), which cannot
sufficiently utilize the global feature. As a result, how to achieve accurate
and scalable certified defense against adversarial patches remains an open
problem.

To this end, we introduce PatchVeto, a detection-based provable defense
against adversarial patches that can flexibly and accurately scale to complex
images and different adversarial patch sizes. PatchVeto is inspired by the
recently-proposed Vision Transformer (ViT) architecture (Dosovitskiy et al.,
2021), which is based on the self-attention mechanism widely used in natural
language processing (Vaswani et al., 2017). PatchVeto is zero-shot (aka.
requires no additional training) by directly reusing a pretrained ViT backbone
for certification.

In a ViT model, the input image is partitioned into small patches. The patches
are fed into several Transformer encoder layers to exchange local information
by attending to each other. The global feature is obtained after the
Transformer layers and used to classify the whole image. Such an attention
mechanism enables a convenient way to exclude local patches from an inference
pass - a patch can be excluded by masking the attentions of other patches
towards it. The remaining patches can still interact with each other to obtain
sufficient global information for prediction.

The certified defense ability of PatchVeto is obtained by manipulating the
attention map of ViT. Given an input image that may contain an adversarial
patch, PatchVeto first makes a prediction using the original ViT, then verify
the prediction with a batch of parallel forwarding passes. Each forwarding
pass uses a different attention mask to exclude certain local patches, and the
masks are designed to guarantee at least one forwarding pass will not attend
to the adversarial patch. The prediction is verified if all masked forwarding
passes produce the same classification.

We evaluate PatchVeto on popular datasets including ImageNet and CIFAR-10 with
adversarial patch sizes ranging from 0.4% to 25%. The results show that
PatchVeto is able to achieve significantly higher clean accuracy and certified
accuracy than existing certified defenses. Specifically, PatchVeto is able to
stably remain a high test accuracy in the verified image subset under
different adversarial patch sizes, although the ratio of verifiable inputs
decreases under larger adversarial patches.

##  2 Background and Related Work

###  2.1 Vision Transformers vs. Convolution Neural Networks

The Vision Transformer (ViT) architecture (Dosovitskiy et al., 2021) is
proposed recently as an alternative to Convolution Neural Networks (CNNs for
short) (Szegedy et al., 2015) in computer vision tasks. ViT is inspired by the
scaling success of self-attention-based architectures (in particular
Transformer (Vaswani et al., 2017)) on NLP tasks. In a ViT, an image is split
into patches and converted to a sequence of linear embeddings of these
patches, which is then fed into a Transformer as the input. Image patches are
treated the same way as tokens (words) in an NLP application. By pre-training
the model on large amounts of data, it is able to attain excellent results as
compared to convolutional networks. Later studies have found that the
Transformer architecture can outperform CNN as the backbone network on most
downstream CV tasks (Liu et al., 2021).

Splitting the input image into patches is not new in ViT. (Brendel & Bethge,
2019) introduced a variant of the ResNet architecture (He et al., 2016) called
BagNet, which classifies an image based on the occurrences of small local
image features. BagNet can achieve reasonable accuracy on ImageNet, although
the accuracy is inferior to ViT due to the lack of model interactions between
the patches. Unlike most existing certified defenses against adversarial
patches that are based on BagNet, ours is based on ViT.

###  2.2 Adversarial Patches vs. Adversarial Perturbations

DNNs are widely known to be vulnerable against adversarial attacks (Szegedy et
al., 2014; Goodfellow et al., 2015). Classic adversarial attacks are known as
adversarial perturbations, which aim to generate a perturbation added to the
pixels of an image that can lead to misclassification.

An adversarial patch is a specicial case of adversarial attack where the pixel
modification is performed within a restricted region in the image. Brown et
al. (2017) first demonstrated the feasibility of fooling an image classifier
with a specifically crafted physical patch. Numerous other approaches (Karmon
et al., 2018; Eykholt et al., 2018) have been proposed afterward to achieve
more practical and effective patch attacks.

Whole-image perturbations, as argued in Chiang et al. (2020), are unrealistic
in the real world, while patch attack scenario is more reasonable for modeling
physical adversary (Eykholt et al., 2018).

###  2.3 Certified Detection vs. Certified Recovery

There are lots of techniques proposed to defend against adversarial patches,
including heuristic defenses and certified defenses. Heuristic defenses
(Naseer et al., 2019; Rao et al., 2020) are mostly based on the empirical
understanding of existing attacks, which may fail on strong adaptive attacks
(Chiang et al., 2020). Certified defenses, instead, aim to rigorously
guarantee the robustness of a given model to patch attacks. Due to the ability
to prevent future adaptive adversaries, certified defenses have attracted many
researchers’ interests in recent years.

The certified defenses can further be classified as certified recovery (Chiang
et al., 2020; Xiang et al., 2021; Levine & Feizi, 2020; Zhang et al., 2020;
Metzen & Yatsura, 2021) and certified detection (McCoyd et al., 2020; Xiang &
Mittal, 2021; Han et al., 2021). Certified recovery aims to recover the
correct prediction on images with adversarial patches, while certified
detection aims to detect adversarial images with no false negative.

Although certified recovery is usually believed to be more advanced than
certified detection, we argue that certified detection may be more practical.
First, recovering the correct prediction under the presence of adversarial
patches is difficult or even impossible in some cases, e.g. when the critical
area of interest (AOI) is hidden by an adversarial patch. Second, optimizing
the model for both clean and adversarial inputs would inevitably decrease the
model’s accuracy in all existing certified recovery approaches, which may harm
the user experience in normal settings. The focus of this paper is certified
detection.

##  3 Problem Setup

Our work shares the same threat model as existing certified defenses against
adversarial patches (Xiang et al., 2021; Levine & Feizi, 2020; Chiang et al.,
2020; Zhang et al., 2020).

The attack and defense are both focused on image classification. We use ${\cal
X}\subset\mathbb{R}^{W\times H\times C}$ to denote the distribution of images
where each image $x\in{\cal X}$ has width $W$, height $H$, number of channels
$C$. We take ${\cal Y}=\\{0,1,\cdots,N-1\\}$ as the label space, where the
number of classes is $N$. We use $f:{\cal X}\rightarrow{\cal Y}$ to denote the
model that takes an image $x\in{\cal X}$ as input and predicts the class label
$y\in{\cal Y}$.

Attacker capability. The attacker can arbitrarily modify pixels within a
restricted region, and this region can be anywhere on the image, even over the
salient object. We assume that all manipulated pixels are within a square
region, and the defender has a conservative estimate (i.e., upper bound) of
the region size. Our method can be generalized to other patch shapes, as long
as the patch can be covered by a restricted rectangle, but we focus on square-
shaped patches for simplicity.

Formally, we assume the attacker can arbitrarily modify an image $x$ within a
constraint set ${\cal A}(x)$. We use a binary pixel block $p\in{\cal
P}\subset\\{0,1\\}^{W\times H}$ to represent the restricted region, where the
pixels within the region are set to $1$. Then, the constraint set ${\cal
A}(x)$ can be expressed as $\\{x^{\prime}=(1-p)\odot x+p\odot
x^{\prime\prime}\\}|x,x^{\prime}\in{\cal
X},x^{\prime\prime}\in\mathbb{R}^{W\times H\times C},p\in{\cal P}\\}$, where
$\odot$ refers to the element-wise product operator, and $x^{\prime\prime}$ is
the content of the adversarial patch.

Attack objective. We focus on the adversarial patch attacks against image
classification models. Given a deep learning model $f$, an image $x$, and its
true class label $y$, the goal of the attacker is to find an image
$x^{\prime}\in{\cal A}(x)\subset{\cal X}$ such that
$f(x^{\prime})=y^{\prime}$, where $y^{\prime}$ is an arbitrary incorrect class
label defined by the attacker and $y^{\prime}\neq y$.

Defense objective. The job of the defender is to design a defended model
$g=(f,v):{\cal X}\rightarrow{\cal Y}\times\\{0,1\\}$, where $g(x)=f(x),v(x)$,
$f(x)\in{\cal Y}$ is the classification result, and $v(x)\in\\{0,1\\}$ is the
verification result indicating whether the prediction $f(x)$ can be verified
($1$ stands for “verified”). For a given input, the defender decides whether
the input can be verified ($v(x)=1$). The verifiable input subset is named as
“robust domain” and noted as ${\cal X}_{\text{trust}}=\\{x\in{\cal X}\ |\
v(x)=1\\}$. The predictions of PatchVeto should be resistant to adversarial
patches in the robust domain, i.e. any adversarial example generated by the
attacker either is ineffective or can be detected (cannot pass the
verification). Formally, for any certified clean data point $x\in{\cal
X}_{\text{trust}}$ and any adversarial example $x^{\prime}\in{\cal A}(x)$, we
ensure either $f(x^{\prime})=f(x)$ or $v(x^{\prime})=0$.

The defender aims to improve the quality of the defended model. The clean
accuracy ${acc}_{\text{clean}}$ (the accuracy of $f$ on the original dataset
${\cal X}$ without considering verification) and the certified accuracy
${acc}_{\text{certified}}$ (the ratio of images in ${\cal X}$ that are
correctly and provably classified) are frequently used in prior work to
describe the performance of certified defenses. Formally,

${acc}_{\text{clean}}=\mathbf{E}_{x\in{\cal X}}[l(f(x),y)]$ (1)
${acc}_{\text{certified}}=\mathbf{E}_{x\in{\cal X}}[l(f(x),y)\ v(x)]$ (2)

where $l$ is the 0/1 error function.

In other words, the defender wants to maximize the ratio of verifiable inputs,
measured by

$r_{\text{trust}}=\frac{|{\cal X}_{trust}|}{|{\cal X}|}$ (3)

and the model accuracy in the trust domain

${acc}_{\text{in-trust}}=\mathbf{E}_{x\in{\cal X}_{trust}}[l(f(x),y)]$ (4)

so that it can retain high test accuracy and guarantee reliability on most
inputs (${\cal X}_{\text{trust}}$), while raising a warning if the input $x$
is potentially malicious ($v(x)=0$). The concept is similar to the selective
prediction approaches that try to integrate a reject option in the neural
network (Geifman & El-Yaniv, 2019). Such a rejection mechanism is helpful in
many perception tasks where a fall-back solution is available. For example, a
driving assistance model can ask the human driver to take over or perform a
conservative operation when it’s not confident about the current situation.

##  4 Our Approach: PatchVeto

We introduce PatchVeto, a detection-based certified defense against
adversarial patch attacks. Our method is developed upon the Vision Transformer
(ViT) architecture, by utilizing its input partition nature and self-attention
mechanism. By simply manipulating the attention masks of ViT and voting over
different masked predictions, PatchVeto can produce high-precision prediction
on the input images and certify robustness on a large portion of them.

###  4.1 The Defended Model

Figure 1: The conceptual model architecture of PatchVeto

The conceptual model architecture of PatchVeto is shown in Figure 1. It can be
viewed as a paired function $g=(f,v)$, where $f$ is a pretrained ViT model for
prediction, and $v$ is a verification function based on the ViT model. Given
an input image $x$, PatchVeto produces a pair $(f(x),v(x))$, where $f(x)$ is
the predicted class of $x$ and $v(x)$ represents whether the prediction is
verified.

In PatchVeto, the input image is splited into non-overlapping patches as in
vanilla ViT models. Suppose the input patch size is $P\times P$ in pixels,
then the input image $x\in{\cal X}\subset\mathbb{R}^{W\times H\times C}$ is
partitioned into a sequence of patches ${\cal P}=\\{p_{i}|i=1,2,...,n\\}$,
where $n=n_{w}\times n_{h}=\frac{W}{P}\times\frac{H}{P}$ is the number of
patches. Each patch $p_{i}$ is then flattened, passed through a linear
projection layer, and added a position embedding to generate a patch embedding
$q_{i}$. A learnable [class] embedding $q_{0}$ is prepended to the sequence of
embeddings, which is used to produce the class prediction after passing
through later layers.

Specifically, the patch embeddings are fed into multiple parallel Transformer
encoder layers ${\cal T}=\\{t_{0},t_{1},...,t_{k}\\}$ to exchange information
between local patches. The encoders share the same weights of the ViT model
$f$, while are used with different attention masks. Each Transformer encoder
$t_{j}$ will produce an encoding of the [class] node, which is then passed
through the MLP head to produce a class prediction $y_{j}$. We call the
predictions produced with the masked attention maps (i.e.
$y_{1},y_{2},...,y_{k}$) as _masked predictions_. The class prediction $f(x)$
is produced by the Transformer encoder without attention masking (i.e.
$y_{0}$), which is equalivent to a direct inference using the base ViT model.
The verification result $v(x)$ is produced by voting over all of the masked
predictions ($y_{1}$ to $y_{k}$). We ensure at least one of the masked
predictions is benign (i.e. can completely mask the adversarial patch out),
which will veto the adversary’s target output even if all other predictions
are compromised. The prediction is verified if all Transformer encoders reach
consensus (i.e. agreeing on the same class prediction).

For example, as illustrated in Figure 1, suppose we use a ViT model with
30$\times$30 input resolution and 10$\times$10 input patch resolution as the
base model. An input image will be partitioned into 3$\times$3 patches. To
defend against adversarial patches with 5$\times$5 resolution, we let each
mask to exclude 2$\times$2 local patches, i.e. a square region with
20$\times$20 resolution. By sliding the 2$\times$2 mask over all local patches
in the image, we can get four (2$\times$2) possible mask locations and
guarantee at least one of the mask locations can completely hide the
5$\times$5-pixel adversarial patch. A certified prediction is produced if the
four masked predictions vote for the same class as the non-masked prediction.

###  4.2 Masking Strategy

(a) $0<W_{adv}<P$

(b) $P<W_{adv}<2P$

(c) $2P<W_{adv}<3P$

Figure 2: Number of ViT patches affected by different sizes of adversarial
patches. $P$ and $W_{adv}$ represent the widths of the ViT input patch and
adversarial patch respectively.

The certification ability of PatchVeto is achieved by ensuring that at least
one of the masked predictions is not influenced by the adversarial patch. The
key to design the attention masks is to determine how many local input patches
might be tainted by the adversarial patch (i.e. containing pixels belonging to
the adversarial patch). Since the input partition plan is fixed and the
adversarial patch is at an arbitrary location in the image, we need to
consider the worst case, i.e. the maximum number of input patches that may be
tainted by the adversary.

Our masking strategy is based on the observation that, _in an image that is
partitioned into non-overlapping $P\times P$-pixel patches, an adversarial
patch with $W_{adv}\times H_{adv}$ resolution can affect at most $N_{W}\times
N_{H}$ input patches, where $N_{W}=\lceil\frac{W_{adv}}{P}\rceil+1$ and
$N_{H}=\lceil\frac{H_{adv}}{P}\rceil+1$._ For example, as illustrated in
Figure 2, if the width $W_{adv}$ of a square adversarial patch is smaller than
the input patch width $P$, four input patches may be tainted (if the
adversarial patch is at the joint of 2$\times$2 input patches). Similarly, a
square adversarial patch with $W_{adv}\in(2P,3P)$ may affect at most
4$\times$4 input patches.

Specifically, given the maximum rectangle shape $W_{adv}\times H_{adv}$ of the
adversarial patch, we can compute the minimum size $N_{W}\times N_{H}$ for the
attention mask. By sliding the mask over the whole input patch grid with a
stride of 1, we can enumerate all $k$ possible locations of the mask, where

$k=(\frac{W}{P}-N_{W}+1)\times(\frac{H}{P}-N_{H}+1)$ (5)

We can guarantee that at least one of the $k$ masks can cover the arbitrary
adversarial patch.

###  4.3 Certification Analysis

In this subsection, we prove that PatchVeto can achieve the defender’s
objective described in Section 3.

After obtaining the regular prediction of the ViT base model $f(x)$, the
verification result of PatchVeto is obtained by

$\displaystyle v(x)\triangleq\left\\{\begin{array}[]{ll}1,&\hbox{if
$f_{1}(x)=f_{2}(x)=...=f_{k}(x)=f(x)$;}\\\
0,&\hbox{otherwise}\end{array}\right.$ (8)

where $f_{j}(x)$ represents the prediction obtained by the ViT base model with
the $j$-th mask position on the attention map.

For any verified clean data point $x\in{\cal X}_{trust}$ and any adversarial
example $x^{\prime}\in{\cal A}(x)$, we need to ensure the adversarial patch is
either ineffective or can be detected. Specifically, assuming $x^{\prime}$ can
pass the verification, we have

$f_{1}(x^{\prime})=f_{2}(x^{\prime})=...=f_{k}(x^{\prime})=f(x^{\prime})$ (9)

Based on our masking strategy (Section 4.2), at least one of the masked
predictions will exclude the adversarial patch in $x^{\prime}$, i.e.

$\exists j\in\\{1,2,...,k\\},\ \textit{s.t.}\ f_{j}(x^{\prime})=f_{j}(x)$ (10)

Since $x$ is a verified input, we have $f(x)=f_{j}(x)$, and thus

$f(x^{\prime})=f_{j}(x^{\prime})=f_{j}(x)=f(x)$ (11)

meaning the adversarial patch attack on $x^{\prime}$ is ineffective.
Therefore, we can guarantee that any effective adversarial patch added to a
trusted input image would not pass the verification.

##  5 Evaluation

We evaluated PatchVeto on both CIFAR-10 (Krizhevsky, 2009) and ImageNet
(Krizhevsky et al., 2012) datasets under different adversarial patch sizes.
The PatchVeto implementation used for evaluation was based on a pretrained
ViT-Base model variant with $16\times 16$ input patch size (ViT-B/16)
(Wightman, 2019), which achieves 81.8% test accuracy on ImageNet. When testing
on CIFAR-10, the ViT-B/16 base model is fine-tuned on the resized CIFAR-10
dataset first to reach 98.7% test accuracy. The base ViT model can also be
replaced with other model variants. All the experiments were conducted on a
Linux server with 4 Nvidia V100 GPU nodes.

Baselines. Most existing approaches on certified adversarial patch defense are
focused on “certified recovery” (as described in Section 2). We compared our
approach PatchVeto (PV for short) with the well-known and/or state-of-the-art
defenses, including recovery-based defenses like Interval Bound Propogation
(IBP) (Chiang et al., 2020), (De) randomized Smoothing (DS) (Levine & Feizi,
2020), PatchGuard (PG) (Xiang et al., 2021), and BagCert (BC) (Metzen &
Yatsura, 2021). Detection-based methods include Minority Report (MR) (McCoyd
et al., 2020), PatchGuard++ (PG++) (Xiang & Mittal, 2021) and (Han et al.,
2021), whose certified detection is also achieved by voting over multiple
inferences. PG++ is based on the BagNet architecture (Brendel, 2020), which
cannot reach optimal accuracy due to the lack of global feature. MR is based
on directly training (fine-tuning) a CNN model to classify the images with a
square region occluded, thus can achieve higher clean and certified accuracy.
However, the original design of MR needs to enumerate a lot of occlusion
positions, which is computationally intensive for high-resolution images. Thus
we additionally implement a more advanced version of MR, named MR+, by porting
our masking strategy to reduce its number of occlusion positions to the same
as in Equation 5. We also replaced the CNN backbone of MR+ to a pretrained
ResNet50 (Wightman, 2019) that has similar performance (81.1% accuracy on
ImageNet) as the ViT backbone used in PatchVeto. Unlike our approach that
requires no training, MR+ needs to fine-tune the model for each adversarial
patch size. Unless otherwise noted, MR+ is fine-tuned for 5 epochs (around 4
hours) on the default training split for each adversarial patch size.

###  5.1 Defense Effectiveness

Method | ImageNet | CIFAR-10
---|---|---
(2% patch size) | (2.4% patch size)
${acc}_{\text{clean}}$ | ${acc}_{\text{certified}}$ | ${acc}_{\text{clean}}$ | ${acc}_{\text{certified}}$
Recovery | IBP (Chiang et al., 2020) | N/A | 47.8 | 30.8
DS (Levine & Feizi, 2020) | 44.4 | 14.0 | 83.9 | 56.2
PG (Xiang et al., 2021) | 43.6 | 15.7 | 84.6 | 57.7
BC (Metzen & Yatsura, 2021) | 45.2 | 22.9 | 86.0 | 60.0
Detection | MR (McCoyd et al., 2020) | N/A | 78.8 | 77.6
90.6 | 62.1
92.4 | 43.8
PG++ (Xiang & Mittal, 2021) | 62.9 | 28.0 | 84.8 | 68.9
62.7 | 32.0 | 82.5 | 71.7
62.1 | 35.5 | 80.2 | 74.3
60.9 | 39.0 | 78.0 | 76.3
SC (Han et al., 2021) | 58.5 | 55.4 | 78.9 | 75.3
MR+ | 75.5 | 56.3 | 97.7 | 83.3
| PV (ours) | 81.8 | 67.2 | 98.7 | 84.1
Table 1: The clean and certified accuracy of different certified defenses on
ImageNet and CIFAR-10. The numbers of IBP, DS, PG, BC, MR, PG++, and SC are
directly copied from their paper. Note that the results of IBP and MR on
ImageNet are not available, because they are computationally intensive or even
infeasible on high-resolution images.

We first compared our approach with existing certified defense approaches in
terms of clean accuracy and certified accuracy. As shown in Table 1, our
approach was able to significantly outperform the other approaches with a
clean accuracy of 81.8% and a certified accuracy of 67.1% on ImageNet with
32$\times$32-pixel adversarial patches. Due to the design of PatchVeto, its
clean accuracy remains the same as the base ViT model, which can be further
improved by using better base models.

The results of detection-based approaches (including ours) are not directly
comparable with the recovery-based approaches because they are designed for
different goals. However, we can notice that our approach was able to achieve
a much higher certified accuracy than recovery-based methods, thus it may be
more practical to use in real world. The main reason is that the certified
detection is based on voting over predictions with small number of patches
excluded, which can still provide sufficient global features, rather than in
recovery-based approaches where each voter is based on a small local patch.

As compared to other certified detection approaches, PatchVeto could achieve
higher clean accuracy and certified accuracy, especially on the ImageNet
dataset. This is because of the advantage of self-attention mechanisms in
Vision Transformers, which can better tolerant the absence of individual local
patches by simply not attending to them. MR+ or other training-based
approaches would inevitably sacrifice accuracy since the model has to be
fitted on different augmented data.

Meanwhile, MR+ requires to be fine-tuned on an augmented dataset for each
adversarial patch size. We have tested the performance of MR+ on ImageNet
under 2%-pixel patches with number of epoches ranging from 3 to 24, which took
2 to 16 hours for training. The clean accuracy and certified accuracy
increased slowly from 72.9% and 52.8% to 78.1% and 61.1% respectively, which
are still lower than the results of PatchVeto achieved with no training
effort.

Max Adv Patch Size | ${acc}_{\text{clean}}$ | ${acc}_{\text{certified}}$ | $r_{\text{trust}}$ | ${acc}_{\text{in-trust}}$
---|---|---|---|---
PV | MR+ | PV | MR+ | PV | MR+ | PV | MR+
0.5% | 81.8 | 75.6 | 72.0 | 61.8 | 80.3 | 71.0 | 89.7 | 87.1
2.0% | 81.8 | 75.5 | 67.2 | 56.3 | 73.5 | 63.1 | 91.4 | 89.3
4.6% | 81.8 | 75.5 | 61.9 | 51.2 | 67.0 | 56.6 | 92.4 | 90.6
8.2% | 81.8 | 76.3 | 56.4 | 48.2 | 60.4 | 52.6 | 93.3 | 91.7
12.8% | 81.8 | 76.2 | 50.5 | 43.4 | 53.7 | 47.0 | 94.0 | 92.4
18.4% | 81.8 | 75.8 | 44.1 | 38.8 | 46.7 | 41.6 | 94.6 | 93.1
25.0% | 81.8 | 75.5 | 37.1 | 34.8 | 39.1 | 37.3 | 94.8 | 93.2
Table 2: The clean accuracy and certified accuracy achieved by PatchVeto on
ImageNet under different adversarial patch sizes.

We further tested the defense effectiveness of PatchVeto and MR+ under
different adversarial patch sizes. The results are shown in Table 2. To
support the defense against different adversarial patch sizes, MR+ requires
fine-tuning the model for each size, while PatchVeto only needs to slightly
adjust the masking strategy, including the size of masks and the number of
mask positions as described in Section 4.2.

The results show that the certified accuracy of PatchVeto dropped from 72.0%
to 37.1% as the adversarial patch size increased from 0.5% to 25.0%, which is
an intuitive result since the stronger adversary (with the ability to modify a
larger region in the image) is harder to defend. Meanwhile, it is interesting
to notice that the accuracy of PatchVeto in the trust domain ($acc_{\text{in-
trust}}$) remained high (even slightly increases) under larger adversarial
patch sizes. This means that PatchVeto can retain a high usability under
strong adversaries - even though PatchVeto may raise more warnings (i.e.
report more unverifiable input images) when defending against stronger
adversaries, it can still promise a high accuracy when it gives verified
predictions.

###  5.2 Ablation Study

Figure 3: The clean and certified accuracy of PV and MR+ on CIFAR-10 with or
without fine-tuning.

PatchVeto does not require fine-tune the model, while MR+ requires fine-tuning
by default. To understand the effect of fine-tuning for both PatchVeto and
MR+, we tested them by adding/removing the fine-tuning phase. The results on
CIFAR-10 are visualized in Figure 3. By adding a four-hour fine-tuning phase
to PatchVeto for each adversarial patch size, the clean accuracy was almost
not changed, and the certified accuracy was slightly increased. For example,
the certified accuracy of PatchVeto on CIFAR-10 with 2.4%-pixel adversarial
patch was increased from 84.1% to 88.3%. In practice, using PatchVeto in the
zero-shot manner (without fine-tuning) is already effective enough. However,
if we remove the fine-tuning process in MR+ (i.e. directly using the original
model for occluded image classification), the certified accuracy would drop
significantly (from 83.3% to 49.5% on CIFAR-10).

The certified accuracy of MR+ became higher than PatchVeto under larger
adversarial patches, e.g. in Figure 3, the certified accuracies of MR+ and PV
were 65.6% and 48.5% respectively under 25%-pixel adversarial patches. This
means that ViT may be less effective than fine-tuned CNN when a large portion
of local features are missing. We further tested the performance of MR+ on
ImageNet under 2%-pixel patches with the number of epochs ranging from 3 to
24, which took 2 to 16 hours for training. The clean accuracy and certified
accuracy increased slowly from 72.9% and 52.8% to 78.1% and 61.1%
respectively, which were still lower than the results of PatchVeto achieved
with no training effort. This again demonstrates the robustness of ViT
architectures to tolerant the absence of a small portion of local features.

We also evaluated the performance of PatchVeto with other ViT variants,
including a more accurate backbone ViT_l16_224 (${acc}_{\text{clean}}=83.1\%$)
and a more coarse-grained ViT_b32_384 (${acc}_{\text{clean}}=81.7\%$). The
result is shown in Figure 5. The certified accuracy remains high for different
ViT backbones, and stronger backbones produce slightly higher certified
accuracy as expected.

###  5.3 Overhead

Figure 4: The certified accuracy of PatchVeto on ImageNet with different ViT
variants.

Figure 5: The per-sample verification latency of PatchVeto and MR+ on
ImageNet.

Since the verification part of PatchVeto is based on a big batch of masked
predictions which might be time-consuming, we further measured the latency of
PatchVeto to verify an input image against different adversarial patch sizes
on a V-100 GPU. As shown in Figure 5, the verification latency is around 240
ms per sample to defend against 0.5%-pixel patches (which requires to
enumerate $k=169$ possible mask positions) and 126 ms for 25%-pixel patches
($k=49$). The verification latency is obviously higher than the regular
inference latency of ViT (around 12 ms per sample), also higher than MR+ with
the ResNet50 backbone. However, we think the latency is still acceptable given
the great benefits of zero-shot certification it provides.

##  6 Conclusion and Future Work

In this paper, we presented a simple yet effective certified defense against
adversarial patches, by utilizing the novel connection between adversarial
patches and input patches in Vision Transformers. We demonstrated the
effectiveness and practicality of the defense on ImageNet and CIFAR-10, as
well as its flexibility to support different sizes of adversarial patches. The
current form of this defense is only designed for a single square adversarial
patch, while other shapes of patches and multiple patches are not handled yet.
A problem with multiple adversarial patches might be that the number of
possible mask positions may be too large, in order to ensure at least one mask
can hide the adversarial patches. More advanced masking strategies and voting
mechanisms might be needed in the future to tackle this problem.

## References

  * Brendel (2020) Wieland Brendel.  Pretrained bag-of-local-features neural networks.  https://github.com/wielandbrendel/bag-of-local-features-models, 2020\.
  * Brendel & Bethge (2019) Wieland Brendel and Matthias Bethge.  Approximating CNNs with bag-of-local-features models works surprisingly well on ImageNet.  In _7th International Conference on Learning Representations (ICLR)_ , 2019.
  * Brown et al. (2017) Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer.  Adversarial patch.  In _Conference on Neural Information Processing Systems Workshops (NeurIPS Workshops)_ , 2017.
  * Chiang et al. (2020) Ping-Yeh Chiang, Renkun Ni, Ahmed Abdelkader, Chen Zhu, Christoph Studor, and Tom Goldstein.  Certified defenses for adversarial patches.  In _8th International Conference on Learning Representations (ICLR)_ , 2020.
  * Chung et al. (2017) Dahjung Chung, Khalid Tahboub, and Edward J Delp.  A two stream siamese convolutional neural network for person re-identification.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 1983–1991, 2017.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _9th International Conference on Learning Representations (ICLR)_ , 2021.  URL https://openreview.net/forum?id=YicbFdNTTy.
  * Eykholt et al. (2018) Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song.  Robust physical-world attacks on deep learning visual classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1625–1634, 2018.
  * Geifman & El-Yaniv (2019) Yonatan Geifman and Ran El-Yaniv.  Selectivenet: A deep neural network with an integrated reject option.  In _International Conference on Machine Learning_ , pp. 2151–2159. PMLR, 2019.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In _3rd International Conference on Learning Representations (ICLR)_ , 2015.
  * Han et al. (2021) Husheng Han, Kaidi Xu, Xing Hu, Xiaobing Chen, Ling Liang, Zidong Du, Qi Guo, Yanzhi Wang, and Yunji Chen.  Scalecert: Scalable certified defense against adversarial patches with sparse superficial layers.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 770–778, 2016.
  * Karmon et al. (2018) Danny Karmon, Daniel Zoran, and Yoav Goldberg.  LaVAN: Localized and visible adversarial noise.  In _Proceedings of the 35th International Conference on Machine Learning (ICML)_ , pp. 2512–2520, 2018.
  * Komkov & Petiushko (2021) Stepan Komkov and Aleksandr Petiushko.  Advhat: Real-world adversarial attack on arcface face id system.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , pp. 819–826. IEEE, 2021.
  * Krizhevsky (2009) Alex Krizhevsky.  Learning multiple layers of features from tiny images.  https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf, 2009\.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  ImageNet classification with deep convolutional neural networks.  In _Conference on Neural Information Processing Systems (NeurIPS)_ , pp. 1106–1114, 2012.
  * Levine & Feizi (2020) Alexander Levine and Soheil Feizi.  (De)randomized smoothing for certifiable defense against patch attacks.  In _Conference on Neural Information Processing Systems, (NeurIPS)_ , 2020.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  _arXiv preprint arXiv:2103.14030_ , 2021.
  * McCoyd et al. (2020) Michael McCoyd, Won Park, Steven Chen, Neil Shah, Ryan Roggenkemper, Minjune Hwang, Jason Xinyu Liu, and David A. Wagner.  Minority reports defense: Defending against adversarial patches.  In _Applied Cryptography and Network Security Workshops (ACNS Workshops)_ , volume 12418, pp. 564–582. Springer, 2020.
  * Metzen & Yatsura (2021) Jan Hendrik Metzen and Maksym Yatsura.  Efficient certified defenses against patch attacks on image classifiers.  In _9th International Conference on Learning Representations (ICLR)_ , 2021.  URL https://openreview.net/forum?id=hr-3PMvDpil.
  * Naseer et al. (2019) Muzammal Naseer, Salman Khan, and Fatih Porikli.  Local gradients smoothing: Defense against localized adversarial attacks.  In _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pp. 1300–1307, 2019.
  * Papernot et al. (2018) Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P Wellman.  Sok: Security and privacy in machine learning.  In _2018 IEEE European Symposium on Security and Privacy (EuroS &P)_, pp. 399–414, 2018.
  * Rao et al. (2020) Sukrut Rao, David Stutz, and Bernt Schiele.  Adversarial training against location-optimized adversarial patches.  In _European Conference on Computer Vision Workshops (ECCV Workshops)_ , 2020.
  * Sun et al. (2016) Yi Sun, Xiaogang Wang, and Xiaoou Tang.  Sparsifying neural network connections for face recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 4856–4864, 2016.
  * Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  In _2nd International Conference on Learning Representations (ICLR)_ , 2014.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 1–9, 2015.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , pp. 6000–6010, 2017.
  * Wei et al. (2019) Jian Wei, Jianhua He, Yi Zhou, Kai Chen, Zuoyin Tang, and Zhiliang Xiong.  Enhanced object detection with deep convolutional neural networks for advanced driving assistance.  _IEEE Transactions on Intelligent Transportation Systems_ , 2019.
  * Wightman (2019) Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * Xiang & Mittal (2021) Chong Xiang and Prateek Mittal.  Patchguard++: Efficient provable attack detection against adversarial patches.  _arXiv preprint arXiv:2104.12609_ , 2021.
  * Xiang et al. (2021) Chong Xiang, Arjun Nitin Bhagoji, Vikash Sehwag, and Prateek Mittal.  Patchguard: A provably robust defense against adversarial patches via small receptive fields and masking.  In _30th USENIX Security Symposium (USENIX Security 21)_. USENIX Association, August 2021.  URL https://www.usenix.org/conference/usenixsecurity21/presentation/xiang.
  * Zhang et al. (2020) Zhanyuan Zhang, Benson Yuan, Michael McCoyd, and David Wagner.  Clipped bagnet: Defending against sticker attacks with clipped bag-of-features.  In _3rd Deep Learning and Security Workshop (DLS)_ , 2020.


# Generatively Augmented Neural Network Watchdog for Image Classification
Networks

Justin M. Bui  Department of Electrical and Computer Engineering
Baylor University
Waco, TX
<EMAIL_ADDRESS>Glauco A. Amigo  Robert J. Marks II, PhD  Department of
Electrical and Computer Engineering
Baylor University
Waco, TX
<EMAIL_ADDRESS>Department of Electrical and Computer Engineering
Baylor University
Waco, TX
<EMAIL_ADDRESS>
###### Abstract

The identification of out-of-distribution data is vital to the deployment of
classification networks. For example, a generic neural network that has been
trained to differentiate between images of dogs and cats can only classify an
input as either a dog or a cat. If a picture of a car or a kumquat were to be
supplied to this classifier, the result would still be either a dog or a cat.
In order to mitigate this, techniques such as the neural network watchdog have
been developed. The compression of the image input into the latent layer of
the autoencoder defines the region of in-distribution in the image space. This
in-distribution set of input data has a corresponding boundary in the image
space. The watchdog assesses whether inputs are in inside or outside this
boundary. This paper demonstrates how to sharpen this boundary using
generative network training data augmentation thereby bettering the
discrimination and overall performance of the watchdog.

##  I Introduction

Neural networks and the techniques associated with machine learning are
readily being used in a variety of applications. In fact, machine learning has
become ubiquitous throughout many industries, including the medical,
automotive, and energy industries [15, 21, 44, 45, 53, 56]. Furthermore, the
development of such techniques has been spurred by the continuous development
and expanded capabilities of tools such as PyTorch and TensorFlow [1, 47].

An increasing need for quality training data goes hand in hand with the
development of new machine learning techniques. Data augmentation and
generation have become a standard practice in the development of deep neural
networks [60, 40, 48, 57, 68]. As systems become more complex, and pipelines
are transitioned from evaluation to production environments, the need for
online out-of-distribution detection increases [22]. One technique to help
mitigate the impact of out-of-distribution inputs is the neural network
watchdog. This paper describes the design and development process for a multi-
tiered neural network watchdog designed to improve the on-line performance of
an image classification neural network. This is achieved by a more effective
sharpening of the boundary of the image space, resulting in more accurate
discrimination of the in-distribution from out-of-distribution data as
dictated by the latent layer of the autoencoder.

##  II Background

One of the challenges associated with the development of classification neural
networks is the detection and classification of out-of-distribution data.
Abbasi et al. [2] provide an excellent introduction to the issues and metrics
associated with out-of-distribution data in classification networks. Out-of-
distribution detection is often times closely coupled to anomaly detection
[16, 23, 42] (a.k.a. novelty filtering [12, 20, 59, 61].) Numerous anomaly
detection techniques have been proposed [67, 49, 54] which demonstrate some
capabilities of detecting out-of-distribution data. One of the more
interesting techniques for anomaly detection is the use of autoencoders [38,
50, 55, 69]. Autoencoders have demonstrated an interesting array of
capabilities, such as implicit learning [61], missing sensor restoration, [41,
62], data denoising [17, 36] and image enhancement [34, 35, 46].

The neural network watchdog [5], is a technique for out-of-distribution
detection. Initially, an autoencoder is used to learn latent representations
of in-distribution data, and then reproduce the original input based on the
representation. As the autoencoder is trained on in-distribution data, the
representations produced by out-of-distribution data result in larger error
during reconstruction. The calculated error is then compared to a threshold
based on the desired level of detection. This threshold operation acts as an
input filter to the original classification or regression network. The
evaluation and selection of an appropriate error function depends on the input
data type and target system performance. For example, a time-series dataset
may use mean bias error, root mean squared error or unscaled mean bounded
relative absolute error [8] as a metric, whereas image based data may use
structural similarity techniques [65, 66], univariate image difference, or
change vector analysis [4]. Additional error determination methods have been
discussed [7, 31].

Preliminary results have demonstrated successful implementations of neural
network watchdogs [5, 6, 14]. The watchdog has been augmented to a multi-
tiered watchdog [3]. The multi-tiered watchdog uses several different out-of-
distribution detection mechanisms designed in coordination to mitigate some of
the performance tradeoffs associated with watchdog development.

While leveraging multiple detection mechanisms, the watchdog concept can be
further improved using data augmentation techniques. The watchdog relies on
the assessment of the boundary of the latent space in the autoencoder.

Without using an autoencoder, classification boundary points can be found
directly from the trained neural network classifier using neural network
inversion [9], [11], [26, 27, 28, 29, 30, 33] , [43, 51, 63, 64]. Here, a
boundary point location is specified at the output of a trained neural network
and the corresponding input is found.

Data augmentation has moved beyond neural network inversion techniques with
the development of GANs [18]. Training data can now be generated using a
variety of GANs [32, 37, 39]. Papers presented by Creswell et al. [10] and
Goodfellow et al. [19] provide excellent reviews of the capabilities of GANs.
Work done by Zhu et al. and previous work demonstrates the use of GANs in the
development of augmented training data [70, 3].

##  III The Neural Network Watchdog

The neural network watchdog has been successfully demonstrated as a method to
improve classifier performance with mixed-distribution input data [5, 6, 14].
The watchdog first uses a generative component to reconstruct input data, such
as an autoencoder. The reconstruction is then compared to the original input
data using an error function. An application-specific criterion is then used
to determine input validity, and whether or not the original input will be
passed into the classifier. Figure 1 shows an example of how data flows
through a watchdog guarded image classification network.

Figure 1: An example of image data flowing through a watchdog guarded image
classifier.

###  III-A Network Structures

The neural network watchdog can be implemented in a variety of architectures,
including disjointed, symbiotic, and multi-tiered. Examples of these
architectures can be seen in Figures 2, 3, and 4. Discussions about the
differences and trade offs between disjointed, symbiotic and multi-tiered
architectures can be found in previous work [3, 6].

Figure 2: The general structure of a Disjointed Watchdog.

The disjointed watchdog (see Fig. 2) consists of a regenerative autoencoder
and a primary classifying neural network, which are trained independently
using the same training data [5]. The selection of an error function is based
on the desired system performance.

Figure 3: The general structure of a Symbiotic Watchdog.

The symbiotic watchdog (see Fig. 3) consists of a single-input multi-output
hybrid generator/classifier network [6]. The network is trained on a single
dataset, with bias towards classifier or generator as necessary. As seen in
Figure 3, both the generated output and classification are produced
simultaneously. In order to determine input validity, the generated
reconstruction is provided to an error function, and the classification output
is only used when the generated reconstruction meets the determined error
criteria.

Figure 4: The general structure of a Multi-tiered Watchdog Structure

The multi-tiered watchdog (see Fig. 4) is comprised of multiple layers of
outlier detection techniques, such as a combined autoencoder and binary
classifier [3]. With multiple watchdog layers, layers can be individually
tuned in order to achieve the desired results.

###  III-B Watchdog Threshold Functions

Fundamental to the successful implementation of a neural network watchdog is
the selection of an appropriate threshold function. Previous work demonstrates
the use of a root mean squared error based threshold function [3, 5, 6].

###  III-C Generative Augmentation

Data augmentation is a common practice to help improve the accuracy of a
neural network. Data augmentation can occur in a variety of ways, such as
image manipulation and data generation. Previous work [3] uses a GAN to
generate training data to be used with a binary classification layer. The
additional training data can be used a number of ways:

  1. 1.

Increased amount of training examples for unbalanced datasets.

  2. 2.

Training a binary classifier for in and out of distribution detection.

  3. 3.

Create a ${(N+1)}$th class to an existing classifier for unrecognized inputs.

##  IV Results

This paper proposes a multi-tiered image classification watchdog for outlier
detection and removal. The multi-tiered watchdog is added to the input data
pipeline, which in turn feeds into the core classification network. By adding
the watchdog to the data pipeline, out-of-distribution images are identified
and discarded prior to presentation to the classification network. Doing so
decreases computational load and improved mixed-distribution classification
accuracy.

###  IV-A Developing a Multi-tiered Watchdog

A multi-tiered watchdog consists of multiple outlier detection techniques that
are combined sequentially. The first layer is comprised of an autoencoder and
an error function. The second layer is a binary classifier to determine
whether input data is in or out of distribution. The multi-tiered watchdog
output is then fed in to a traditional multi-class convolutional neural
network classifier.

###  IV-B Training and Evaluation Datasets

To evaluate the efficacy of the multi-tiered watchdog, a combination of
datasets will be used to train and evaluate the networks. The German Traffic
Sign Benchmark (_GTSRB_) dataset [58] is used as the primary dataset, and is
considered to be in-distribution. Examples of the GTSRB dataset, images of 43
different traffic signs, can be seen in Figure 5. Approximately 39,000 images
are used to train the watchdog and core classifier with an additional 12,600
images to be used to independently evaluate all of the networks.

Figure 5: Examples from the in-distribution German traffic sign dataset.

To augment the evaluation dataset, 10,000 additional images will be used from
each the CIFAR-10 and CIFAR-100 datasets to create a 32,600 image mixed
distribution dataset. In order to effectively evaluate the networks two series
of labels are used for this dataset. The first is the original image label,
and the second is a binary label corresponding to either in or out of
distribution. These series of labels will allow for an effective evaluation of
the core and binary classifiers. Examples of the CIFAR-10 and CIFAR-100 images
can be seen in Figure 6.

Figure 6: Example images from the CIFAR-10 and CIFAR-100 datasets, which are
added to create a mixed-distribution evaluation dataset.

###  IV-C Training Process

In order to successfully train the multi-tiered watchdog, the training process
is divided up into four sub-processes:

  1. 1.

Train the autoencoder.

  2. 2.

Define generative network parameters.

  3. 3.

Train the binary classifier.

  4. 4.

Train the core classifier.

The core classifier and autoencoder can be trained independently. The
generator and binary classifier are dependent on the performance of the
autoencoder, and therefore must be trained after. Both the autoencoder and the
core classifier will be trained on the in-distribution GTSRB images. The
binary classifier is then trained on a combination of in-distribution GTSRB
images, as well as near boundary images created by the generator [3].

####  IV-C1 Training the Autoencoder

The autoencoder is trained on in-distribution data, with the intent to
regenerate the input with minimal mean squared error. Unlike some datasets,
such as the MNIST or CIFAR datasets, the GTSRB dataset has relatively low
numbers of images per each class. To compensate for this fact, the training
data is augmented using tools such as TensorFlow’s built in Image or Keras
APIs. Such tools allow for traditional image and computer vision processing
techniques, including image rotation, cropping, flip, and grayscale
conversion. This augmentation allows the autoencoder to produce higher quality
results on relatively small training sets.

####  IV-C2 Training the Binary Classifier

A key component of training the binary classifier is the generation of new
data. A generator is designed to create data which closely resembles near-
threshold conditions [3]. The near threshold data is classified as out-of-
distribution for training purposes. Once the parameters have been defined and
additional training data has been generated, the binary classifier is trained
to differentiate between in-distribution and generated out-of-distribution
data. One important note is that as adjustments are made to the autoencoder,
they must also be made to the generator and binary classifier.

####  IV-C3 Training the Core Classifier

The process for training the core classifier is similar to training the
autoencoder. Training data is augmented, as previously mentioned, through the
use of the TensorFlow APIs, and the classifier is trained using categorical
cross entropy training loss.

Figure 7 shows the training accuracy and losses for the classifier.

Figure 7: Classifier training history for validation and training accuracy and
losses. The validation dataset is a subset of the training data, and does not
overlap with the data that will be used for evaluation.

##  V Evaluating The Networks

Once all of the networks have been trained, they are evaluated using the
mixed-distribution dataset. The individual performance of the core classifier,
autoencoder and binary classifier are first individually evaluated. The
autoencoder and binary classifier are then added to the data input pipeline,
and the combined system is evaluated.

###  V-A Unguarded Core Classifier Performance

The performance of the core classifier is evaluated using both the German
traffic sign and mixed-distribution datasets. The isolated performance of the
classifier, as shown in Figure 8, is determined by evaluation using the in-
distribution data, and serves as the performance baseline. The normalized
performance of the core classifier can be seen in Figure 9.

Figure 8: The ROC curves of the core classifier performance evaluated on the
43 independent classes of the _GTSRB_ dataset. Figure 9: The normalized ROC
curve of the core classifier performance on the _GTSRB_ dataset.

In order to understand the impact of out-of-distribution data, the same
evaluation is performed using the mixed-distribution dataset. The online
performance of the core classifier, as shown in Figures 10 and 11,
demonstrates the reduced accuracy due to out-of-distribution data. With a
ratio of ${1/3}$ in-distribution to ${2/3}$ out-of-distribution data, our
normalized accuracy and precision drops to approximately 33%, as expected.

Figure 10: The ROC curves of the core classifier performance evaluated on the
mixed distribution dataset. Figure 11: The normalized ROC curve of core
classifier performance evaluated on the mixed distribution dataset.

###  V-B Autoencoder Performance and Threshold Selection

The performance of the autoencoder is evaluated by measuring the structural
similarity score, defined in Equation 1, of the original and regenerated
images and comparing it to the determined structural similarity threshold
value. Figure 18 shows the ROC curve for binary classification of in-
distribution vs out-of-distributed as across several structural similarity
thresholds.

$SSIM(x,y)=\frac{(2\mu_{x}\mu_{y}+C_{1})+(2\sigma_{xy}+C_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+C_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+C_{2})}$
(1)

The constants $C_{1}$ and $C_{2}$ are defined by Equations 2 and 3, where L
represents the data input range value, and $K_{1}$ and $K_{2}$ are constants
$<<$ 1 [65].

$C_{1}=(K_{1}*L)^{2}$ (2) $C_{2}=(K_{2}*L)^{2}$ (3)

Examples of the autoencoder image regeneration can be seen in Figure 13, along
with their original images in Figure 12 and the structural similarity
difference in Figure 14. The structural similarity scores for these images are
found in Table I.

Figure 12: The original _GTSRB_ sign images. Figure 13: An example of the
autoencoder regenerated _GTSRB_ sign images. Figure 14: The structural
similarity images of the _GTSRB_ examples.

As a reference, the autoencoder regeneration of a sample of CIFAR images is
provided in Figure 16, as well as the respective original images in Figure 15
and the difference in Figure 17.

Figure 15: The original CIFAR images. Figure 16: An example of the autoencoder regenerated CIFAR images. Figure 17: The structural similarity images of the CIFAR examples. TABLE I: Structural Similarity Scores | Image 1 | Image 2 | Image 3 | Image 4
---|---|---|---|---
Signs Scores: | 0.9303 | 0.9715 | 0.9184 | 0.8903
CIFAR Scores: | 0.7488 | 0.7767 | 0.8624 | 0.8318
Figure 18: The ROC curve for the autoencoder’s performance, as a function of
structural similarity score.

Based on these results, and depending on the desired level of protection, a
structural similarity score of between 0.83 and 0.87 is an appropriate
selection. For the remainder of this proof of concept, a structural similarity
score of 0.85 is used.

###  V-C Binary Classifier Data Generation

With the baseline autoencoder trained and the error function established, a
generative network is used to create additional training and evaluation data.
The newly generated data is used to train the binary classifier. The
generative network is designed to generate images with a target structural
similarity score. Based on the previously selected 0.85 score, a target score
of 0.90 will be used. Example generated images may be seen in Figure 19.

Figure 19: Generated binary class images with a target structural similarity
score of 0.90

###  V-D Binary Classifier Performance

The performance of the binary classifier is evaluated on an independent subset
of the generated data. The goal of the binary classifier is to differentiate
between in- and out-of-distribution images that reside close to the dataset
manifold which have been permitted by the autoencoder layer. The ROC curve for
the binary classifier can be seen in Figure 20.

Figure 20: The ROC curve for the binary classifier, evaluated on the generated
test dataset.

###  V-E Final performance of the Guarded Classifier

The efficacy of the multi-tiered watchdog is determined by the combined
performance of the autoencoder and the binary classifier. With the watchdog
fully implemented, the final system performance is evaluated. The ROC curve
seen in Figure 21 shows the watchdog guarded performance on the mixed-
distribution evaluation dataset.

Figure 21: The normalized ROC curve for the watchdog guarded image classifier.

A final comparison of the ROC curves for the unguarded, guarded, and in-
distribution only datasets may be seen in Figure 22. The introduction of the
watchdog results in significantly improved performance when compared to the
unguarded core classifier alone.

Figure 22: The normalized ROC curves for the unguarded, watchdog guarded, and
ideal core classifier.

##  VI Conclusion

A multi-tiered neural network watchdog is developed to improve online
classification neural network performance with respect to out-of-distribution
data. A generative network is trained to produce data which resembles out-of-
distribution data permitted by the autoencoder. A binary classifier is trained
to differentiate between the permitted out-of-distribution data and in-
distribution data. The combination of the autoencoder and binary classifier
results in reliable out-of-distribution detection and filtering. The final
results presented demonstrate classification improvements when evaluated on
mixed-distribution input data.

## References

  * [1] Abadi, Martín, et al. ”Tensorflow: A system for large-scale machine learning.” 12th USENIX symposium on operating systems design and implementation (OSDI 16). 2016.
  * [2] Abbasi, Mahdieh, et al. ”Toward Metrics for Differentiating Out-of-Distribution Sets.” arXiv preprint arXiv:1910.08650 (2019).
  * [3] Amigo Galán, Glauco A., Justin Bui, and Robert J. Marks. ”Cascade Watchdog: A Multi-tiered Adversarial Guard for Outlier Detection.” arXiv preprint arXiv:2108.09375 (2021).
  * [4] Bruzzone, Lorenzo, and Diego F. Prieto. ”Automatic analysis of the difference image for unsupervised change detection.” IEEE Transactions on Geoscience and Remote sensing 38.3 (2000): 1171-1182.
  * [5] Bui, Justin, and Robert J. Marks II. “Autoencoder Watchdog Outlier Detection for Classifiers.” arXiv preprint arXiv:2010.12754 (2020).
  * [6] Bui, Justin, and Robert J. Marks II. ”Symbiotic Hybrid Neural Network Watchdog For Outlier Detection.” arXiv preprint arXiv:2103.00582 (2021).
  * [7] Chalom, Edmond, Eran Asa, and Elior Biton. ”Measuring image similarity: an overview of some useful applications.” IEEE Instrumentation & Measurement Magazine 16.1 (2013): 24-28.
  * [8] Chen, Chao, Jamie Twycross, and Jonathan M. Garibaldi. ”A new accuracy measure based on bounded relative error for time series forecasting.” PloS one 12.3 (2017): e0174202.
  * [9] Cohn, D., L.E. Atlas, R. Ladner, M.A. El-Sharkawi, R.J. Marks II, M.E. Aggoune, D.C. Park “Training connectionist networks with queries and selective sampling,” Advances in Neural Network Information Processing Systems 2, Morgan Kaufman Publishers, Inc., San Mateo, CA. 1990.
  * [10] Creswell, Antonia, et al. ”Generative adversarial networks: An overview.” IEEE Signal Processing Magazine 35.1 (2018): 53-65.
  * [11] Duren, Russell W., Robert J. Marks II, Paul D. Reynolds and Matthew L. Trumbo “Real-Time Neural Network Inversion on the SRC-6e Reconfigurable Computer,” IEEE Transactions on Neural Networks, vol. 18, no. 3, May 2007 pp. 889-901.
  * [12] El-Sharkawi, M.A., R.J. Marks II, Robert J. Streifel and I. Kerszenbaum “Detection and Localization of Shorted-Turns in the DC-Field Winding of Turbine-Generator Rotors Using Novelty Filters and Fuzzified Neural Networks,” in Fuzzy System Theory in Electrical Power Engineering, M.E. El-Hawary, editor (IEEE Press, 1998), pp.85-111.
  * [13] Ferwerda, James A., and Fabio Pellacini. ”Functional difference predictors (FDPs): measuring meaningful image differences.” ASILOMAR CONFERENCE ON SIGNALS SYSTEMS AND COMPUTERS. Vol. 2. IEEE; 1998, 2003.
  * [14] Fredieu, C. Tanner, et al. ”Classification of Common Waveforms Including a Watchdog for Unknown Signals.” arXiv preprint arXiv:2108.07339 (2021).
  * [15] Garg, Arunim, and Vijay Mago. ”Role of machine learning in medical research: A survey.” Computer Science Review 40 (2021): 100370.
  * [16] Golan, Izhak, and Ran El-Yaniv. ”Deep anomaly detection using geometric transformations.” arXiv preprint arXiv:1805.10917 (2018).
  * [17] Gondara, Lovedeep. ”Medical image denoising using convolutional denoising autoencoders.” 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW). IEEE, 2016.
  * [18] Goodfellow, Ian, et al. ”Generative adversarial nets.” Advances in neural information processing systems 27 (2014).
  * [19] Goodfellow, Ian, et al. ”Generative adversarial networks.” Communications of the ACM 63.11 (2020): 139-144.
  * [20] Guttormsson, S., R.J. Marks II, M.A. El-Sharkawi and I. Kerszenbaum “Elliptical novelty grouping for on-line short-turn detection of excited running rotors,” IEEE Transactions on Energy Conversion, IEEE Transactions on Volume: 14 1 , March 1999, pp. 16 -22.
  * [21] Hanga, Khadijah M., and Yevgeniya Kovalchuk. ”Machine learning and multi-agent systems in oil and gas industry applications: A survey.” Computer Science Review 34 (2019): 100191.
  * [22] Haug, Samuel, Robert J. Marks, and William A. Dembski. ”Exponential Contingency Explosion: Implications for Artificial General Intelligence.” IEEE Transactions on Systems, Man, and Cybernetics: Systems (2021).
  * [23] Hendrycks, Dan, Mantas Mazeika, and Thomas Dietterich. ”Deep anomaly detection with outlier exposure.” arXiv preprint arXiv:1812.04606 (2018).
  * [24] Hwang, J.N. C.H. Chan, R.J. Marks II “Frequency selective surface design based on iterative inversion of neural networks,” Proceedings of the International Joint Conference on Neural Networks, San Diego, 17-21 June 1990, vol. I, pp.I39-I44.
  * [25] Hwang, J.N., J.J. Choi, S. Oh, R.J. Marks II “Query learning based on boundary search and gradient computation of trained multilayer perceptrons,” Proceedings of the International Joint Conference on Neural Networks, San Diego, June, 1990, 17-21 June 1990, vol. III, pp.III57-III62.
  * [26] Hwang, J.N. C.H. Chan, R.J. Marks II “Frequency selective surface design based on iterative inversion of neural networks,” Proceedings of the International Joint Conference on Neural Networks, San Diego, 17-21 June 1990, vol. I, pp.I39-I44.
  * [27] Hwang, J.N., J.J. Choi, S. Oh, R.J. Marks II “Query learning based on boundary search and gradient computation of trained multilayer perceptrons,” Proceedings of the International Joint Conference on Neural Networks, San Diego, June, 1990, 17-21 June 1990, vol. III, pp.III57-III62.
  * [28] Jensen, Craig A., Russell D. Reed, Mohamed A. El-Sharkawi, Robert J. Marks II “Location of Operating Points on the Dynamic Security Border Using Constrained Neural Network Inversion,” Proceedings of the International Conference on Intelligent Systems Applications to Power Systems (ISAP), pp.209-217, Seoul, Korea, July 6-10, 1997\.
  * [29] Jensen, Craig A., Reed, R.D.; Marks, R.J., II; El-Sharkawi, M.A.; Jae-Byung Jung; Miyamoto, R.T.; Anderson, G.M.; Eggen, C.J. “Inversion of feedforward neural networks: algorithms and applications,” Proceedings of the IEEE, Volume: 87 9, Sept. 1999, Page(s): 1536 -1549
  * [30] Jensen, Craig A., M.A. El-Sharkawi and R.J. Marks II “Power Security Boundary Enhancement Using Evolutionary-Based Query Learning,” Engineering Intelligent Systems, vol.7, no.9, pp.215-218 (December 1999).
  * [31] Johnson, Garrett M., and Mark D. Fairchild. ”Measuring images: Differences, quality, and appearance.” Human vision and electronic imaging VIII. Vol. 5007. International Society for Optics and Photonics, 2003.
  * [32] Karras, Tero, Samuli Laine, and Timo Aila. ”A style-based generator architecture for generative adversarial networks.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
  * [33] Kassabalidis, I.N., Mohamed El-Sharkawi, Robert J. Marks II “Border Identification For Power System Security Assessment Using Neural Network Inversion: An Overview,” 2002 Congress on Evolutionary Computation, 2002 IEEE World Congress on Computational Intelligence May 12-17, 2002, Honolulu, pp.1075-1079.
  * [34] Kim, Juhwan, Seokyong Song, and Son-Cheol Yu. ”Denoising auto-encoder based image enhancement for high resolution sonar image.” 2017 IEEE Underwater Technology (UT). IEEE, 2017.
  * [35] Lore, Kin Gwn, Adedotun Akintayo, and Soumik Sarkar. ”LLNet: A deep autoencoder approach to natural low-light image enhancement.” Pattern Recognition 61 (2017): 650-662.
  * [36] Lu, Xugang, et al. ”Speech enhancement based on deep denoising autoencoder.” Interspeech. 2013.
  * [37] Mao, Xudong, et al. ”Least squares generative adversarial networks.” Proceedings of the IEEE international conference on computer vision. 2017.
  * [38] Marchi, Erik, et al. ”A novel approach for automatic acoustic novelty detection using a denoising autoencoder with bidirectional LSTM neural networks.” 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015.
  * [39] Metz, Luke, et al. ”Unrolled generative adversarial networks.” arXiv preprint arXiv:1611.02163 (2016).
  * [40] Mikołajczyk, Agnieszka, and Michał Grochowski. ”Data augmentation for improving deep learning in image classification problem.” 2018 international interdisciplinary PhD workshop (IIPhDW). IEEE, 2018.
  * [41] Narayanan, Sreeram, R.J. Marks II , John L. Vian, J.J. Choi, M.A. El-Sharkawi and Benjamin B. Thompson “Set Constraint Discovery: Missing Sensor Data Restoration Using Auto-Associative Regression Machines,” Proceedings of the 2002 International Joint Conference on Neural Networks, 2002 IEEE World Congress on Computational Intelligence, May12-17, 2002, Honolulu, pp. 2872-2877.
  * [42] Nguyen, Duc Tam, et al. ”Anomaly detection with multiple-hypotheses predictions.” International Conference on Machine Learning. PMLR, 2019.
  * [43] Oh, Seho, R.J. Marks II and M.A. El-Sharkawi “Query based learning in a multilayered perceptron in the presence of data jitter,” Applications of Neural Networks to Power Systems, (Proceedings of the First International Forum on Applications of Neural Networks to Power Systems), July 23-26, 1991, Seattle, WA, (IEEE Press, pp.72-75).
  * [44] Ozbayoglu, Ahmet Murat, Mehmet Ugur Gudelek, and Omer Berat Sezer. ”Deep learning for financial applications: A survey.” Applied Soft Computing 93 (2020): 106384.
  * [45] Paleyes, Andrei, Raoul-Gabriel Urma, and Neil D. Lawrence. ”Challenges in deploying machine learning: a survey of case studies.” arXiv preprint arXiv:2011.09926 (2020).
  * [46] Park, Seonhee, et al. ”Dual autoencoder network for retinex-based low-light image enhancement.” IEEE Access 6 (2018): 22084-22093.
  * [47] Paszke, Adam, et al. ”Pytorch: An imperative style, high-performance deep learning library.” Advances in neural information processing systems 32 (2019): 8026-8037.
  * [48] Perez, Luis, and Jason Wang. ”The effectiveness of data augmentation in image classification using deep learning.” arXiv preprint arXiv:1712.04621 (2017).
  * [49] Pradhan, Manoranjan, Sateesh Kumar Pradhan, and Sudhir Kumar Sahu. ”Anomaly detection using artificial neural network.” International Journal of Engineering Sciences & Emerging Technologies 2.1 (2012): 29-36.
  * [50] Qi, Yumei, et al. ”Stacked sparse autoencoder-based deep network for fault diagnosis of rotating machinery.” Ieee Access 5 (2017): 15066-15079.
  * [51] Reed, Russell D. and Robert J. Marks II “An Evolutionary Algorithm for Function Inversion and Boundary Marking,” Proceedings of the IEEE International Conference on Evolutionary Computation, p. 794-797, Perth, Australia. November 26-30, 1995.
  * [52] Rosin, Paul L., and Tim J. Ellis. ”Image difference threshold strategies and shadow detection.” BMVC. Vol. 95. 1995.
  * [53] Rabe, Martin, Stefan Milz, and Patrick Mader. ”Development Methodologies for Safety Critical Machine Learning Applications in the Automotive Domain: A Survey.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
  * [54] Sabokrou, Mohammad, et al. ”Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes.” Computer Vision and Image Understanding 172 (2018): 88-97.
  * [55] Sakurada, Mayu, and Takehisa Yairi. ”Anomaly detection using autoencoders with nonlinear dimensionality reduction.” Proceedings of the MLSDA 2014 2nd workshop on machine learning for sensory data analysis. 2014.
  * [56] Sharp, Michael, Ronay Ak, and Thomas Hedberg Jr. ”A survey of the advancing use and development of machine learning in smart manufacturing.” Journal of manufacturing systems 48 (2018): 170-179.
  * [57] Shorten, Connor, and Taghi M. Khoshgoftaar. ”A survey on image data augmentation for deep learning.” Journal of Big Data 6.1 (2019): 1-48.
  * [58] Stallkamp, Johannes, et al. ”The German traffic sign recognition benchmark: a multi-class classification competition.” The 2011 international joint conference on neural networks. IEEE, 2011.
  * [59] Streifel, R.J., R.J. Marks II, M.A. El-Sharkawi and I. Kerszenbaum “Detection of Shorted-Turns in the Field Winding of Turbine-Generator Rotors Using Novelty Detectors: Development and Field Test,” IEEE Transactions on Energy Conversion, vol.11, no.2, June 1996, pp.312-317.
  * [60] Taylor, Luke, and Geoff Nitschke. ”Improving deep learning with generic data augmentation.” 2018 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 2018.
  * [61] Thompson, Benjamin B., Robert J Marks II , Jai J Choi, Mohamed A El-Sharkawi “Implicit Learning in Autoencoder Novelty Assessment,” Proceedings of the 2002 International Joint Conference on Neural Networks, 2002 IEEE World Congress on Computational Intelligence, May12-17, 2002, Honolulu, pp. 2878-2883.
  * [62] Thompson, Benjamin B., Robert J. Marks II, and Mohamed A. El-Sharkawi “On the Contractive Nature of Autoencoders: Application to Missing Sensor Restoration,” 2003 International Joint Conference on Neural Networks, July 20-24, 2003 , Portland, Oregon (pp. 3011-3016)
  * [63] Tsang, Leung, Z. Chen, S. Oh, R.J. Marks II and A.T.C. Chang “Inversion of snow parameters from passive microwave remote sensing measurements by a neural network trained with a multiple scattering model,” Proceedings of the 1991 International Geoscience and Remote Sensing Symposium, 3-7 June 1991, Espoo, Finland.
  * [64] Tsang, Leung, Z. Chen, S. Oh, R.J. Marks II and A.T.C. Chang “Inversion of snow parameters from passive microwave remote sensing measurements by a neural network trained with a multiple scattering model,” IEEE Transactions on Goescience and Remote Sensing, vol. 30, no.5, pp. 1015-1024 (1992).
  * [65] Wang, Zhou, and Alan C. Bovik. ”Mean squared error: Love it or leave it? A new look at signal fidelity measures.” IEEE signal processing magazine 26.1 (2009): 98-117.
  * [66] Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, Apr. 2004.
  * [67] Wei, Qi, et al. ”Anomaly detection for medical images based on a one-class classification.” Medical Imaging 2018: Computer-Aided Diagnosis. Vol. 10575. International Society for Optics and Photonics, 2018.
  * [68] Wen, Qingsong, et al. ”Time series data augmentation for deep learning: A survey.” arXiv preprint arXiv:2002.12478 (2020).
  * [69] Zhou, Chong, and Randy C. Paffenroth. ”Anomaly detection with robust deep autoencoders.” Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 2017.
  * [70] Zhu, Xinyue, et al. ”Data augmentation in emotion classification using generative adversarial networks.” arXiv preprint arXiv:1711.00648 (2017).