# Generalized Adversarial Examples: Attacks and Defenses

Haojing Shen, xxx, xxx, , xxx Haojing Shen, Sihong Chen, and Xizhao Wang are
with Big Data Institute, College of Computer Science and Software Engineering,
Guangdong Key Lab. of Intelligent Information Processing, Shenzhen University,
Shenzhen 518060, Guangdong, China (Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>).Ran Wang is with the College of
Mathematics and Statistics, Shenzhen University, Shenzhen 518060, China and
also with the Shenzhen Key Laboratory of Advanced Machine Learning and
Applications, Shenzhen University, Shenzhen 518060, China. (e-mail:
wangran@szu.edu.cn).Corresponding author: Xizhao Wang.

###### Abstract

Most of the works follow such definition of adversarial example that is
imperceptible to humans but can fool the deep neural networks (DNNs). Some
works find another interesting form of adversarial examples such as one which
is unrecognizable to humans, but DNNs classify it as one class with high
confidence and adversarial patch. Based on this phenomenon, in this paper,
from the perspective of cognition of humans and machines, we propose a new
definition of adversarial examples. We show that imperceptible adversarial
examples, unrecognizable adversarial examples, and adversarial patches are
derivates of generalized adversarial examples. Then, we propose three types of
adversarial attacks based on the generalized definition. Finally, we propose a
defence mechanism that achieves state-of-the-art performance. We construct a
lossy compression function to filter out the redundant features generated by
the network. In this process, the perturbation produced by the attacker will
be filtered out. Therefore, the defence mechanism can effectively improve the
robustness of the model. The experiments show that our attack methods can
effectively generate adversarial examples, and our defence method can
significantly improve the adversarial robustness of DNNs compared with
adversarial training. As far as we know, our defending method achieves the
best performance even though we do not adopt adversarial training.

###### Index Terms:

Adversarial examples, adversarial attacks, adversarial defenses.

##  I Introduction

Recent deep neural networks (DNNs) make breakthroughs in many areas such as
computer vision, speech recognition and so on. The power of DNNs brings people
infinite reverie. Many significant and interesting works on DNNs bursting out.
There are many kinds of DNNs as such convolution neural network (e.g. LeNet
[1], AlexNet [2], ResNet [3]), recurrent neural network (e.g. LSTM [4]). As
the rapid development of DNNs, more and more people focus on the security of
DNNs. In particular, creating adversarial examples and defending adversarial
attack are crucial techniques in the security of DNNs.

The vulnerability [5] of DNNs receives great attention since it has been
found. Great researches are focusing on this topic in literature. These
researches can be roughly classified into two categories, such as adversarial
attacks [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] and adversarial defences [5,
16, 17, 18, 19, 20, 21, 22]. The methods of adversarial attack always want to
create the adversarial examples which can fool the DNNs. Furthermore, the
adversarial examples are usually imperceptible to a human. Inversely,
adversarial defences make DNNs more robustness to adversarial examples. The
adversarial attacks and adversarial defences just like a zero-sum game.

Many works follow such definition of adversarial example that adversarial
example is very close to original natural images but fool the DNNs [5]. In the
context of this definition, an adversarial example is imperceptible to human
(We call them imperceptible adversarial examples). However, there are some
other forms of adversarial examples. For example, some researches add some
patches to the original image for fooling the DNNs [23, 24, 25, 26]. These
examples are called adversarial patches which can fool the DNNs though they
are perceptive to human. Moreover, the adversarial noise images manipulated
willfully by humans can fool the DNNs [27]. Though these noise images are no
meaning for human, DNNs still classify them as a certain class with high
confidence (we call them as unrecognizable adversarial examples). Therefore,
the definition of adversarial examples, that is imperceptible to human and
fool DNNs is not very suitable.

To solve this problem, this paper represents a novel definition of adversarial
examples which is more generalized than that [5]. We want to look for more
generalized definition which can adaptive current most of the adversarial
examples. Then, based on our generalized definition, we proposed three types
of attacks and a defensive mechanism. In this paper, we mainly focus on
defensive mechanism. Firstly, we present that the adversarial example cannot
change the objective reality since all machine tasks are in the service of
humanity, and humans understand the world according to the objective reality.
For example, as shown in ”B” part of Fig. 1, the left subfigure is an English
word, called ”F”. We assume it is an objective reality (e.g. human and machine
all called it as ”F”). Then we get medial subfigure by adding little pixel in
the left subfigure. These perturbation does not bother humans recognition in
pixel space, but the machine may be confused with other words in some feature
space. We consider it an adversarial example if the model classifies it
wrongly. Furthermore, the right subfigure also made by adding little pixel in
the left subfigure. We find that in pixel space it becomes another word which
is called ”E”. Secondly, it is obvious that not all adversarial examples are
imperceptible to humans such as Adversarial patches and Unrecognizable
adversarial examples. We point out that it is significantly different between
human vision and DNNs to recognize the images. As shown in ”A” part of Fig. 1,
we find that when humans identify clean or disturbing images, the answer is
always unique, whereas machines cannot give a definite answer. If the machine
misidentified the disturbing image, such a sample is known as an adversarial
example. Besides, if an adversarial example is imperceptible to human, the
”imperceptibility” is defined in pixel space. Therefore, we emphasize that the
human recognizes an image in pixel space while the machine does that in a
special space which is a machine ” vision” space (”C” part of Fig. 1). We call
as Feature space. Therefore, we draw two rules about generalized adversarial
examples as followed:

  1. 1.

The adversarial example cannot change the objective reality.

  2. 2.

The adversarial example is imperceptible to the machine in Feature space
rather than to human vision.

According to these two rules, we assume that existing a function which can map
the original pixel space to the feature space. We describe adversarial
examples as those that are imperceptible to human in the feature space and can
fool the DNN. Then, we provided three types of attacks included most of the
current attack methods based on our generalized form of adversarial examples.
Finally, based on the generalized adversarial examples, we propose a defence
mechanism to improve the adversarial robustness of the model. Our defence
strategy will compress the information of input and filter many of that,
including the adversarial perturbation. And we theoretically certify the
effectiveness of the defence under a simple case. Besides, we provide a
certain defensive function and empirically verify the effectiveness of our
defence mechanism. It is noteworthy that our defensive model achieves state-
of-the-art performance.

The contributions are listed as below:

  1. 1.

We draw a novel and more generalized definition of adversarial examples by
keeping the properties of imperception and fooling in a feature space rather
than original space or pixel space since the cognition mechanism of human is
remarkably different with the machine. Due to this manipulation, we give the
attacker more freedom to modify the original image (e.g. adversary can add
small imperceptible perturbation or adversarial patches or even using an
adversarial noise image to instead original image).

  2. 2.

Based on our generalized form of adversarial examples, we provide three types
of adversarial attack to generate imperceptible adversarial examples,
unrecognizable adversarial examples and adversarial patch, which are shown in
Fig. 4. Most of the existed attack methods can be derived from our definition
of adversarial examples.

  3. 3.

According to the generalized form of adversarial examples, we propose a
defensive mechanism against adversarial attacks. We point out that a good
defensive strategy needs to have two properties to prevent the perturbation
from pixel space. Our defence method is simple and significantly improve the
adversarial robustness of the DNNs.

  4. 4.

The attack effectivity of our attack method of generating imperceptible
adversarial examples is better than most of attacks method such as FGSM [6],
L2BIA [7], LBFGS [5] and so on. It drops the accuracy from $98.49\%$ to
$59.59\%$, which is better than CW [10] on MNIST dataset. Moreover, it drops
the accuracy respectively from $88.92\%$ to $46.69\%$ on CIFAR10 and from
$69.60\%$ to $49.07\%$ on ImageNet. Our performance is very close to or even
better than that of PGD [16] and CW [10], which are state-of-the-art attack
method. And even though we do not use adversarial training, our defending
method achieves state-of-the-art performance which defeat all existed methods.

In the rest of this paper, Section II introduce the related work in
literature. In Section III, we define generalized adversarial examples. In
Section IV, we derive three types of adversarial attacks from the definition
of generalized adversarial examples. In Section V, we introduce our
adversarial defence which is state-of-the-art without adversarial training. In
Section VI, we conduct large experiments and confirm that our methods are
effective. Finally, Section VII concludes the paper.

##  II Related works

Figure 1: An overview of . (b) An example of perturbation changing the
objective reality (For better understanding, we exaggerate the effect of the
perturbation). Left: That is a English Word called ”F”. Middle: An image is
generated by combining left picture and small perturbation. The perturbation
does not change the objective reality of left picture. Right: An image is
generated by combining left picture and small perturbation. However, the
perturbation does change the objective reality of the original picture. Figure
2: An overview of our attacks and defending strategy based the definition of
generalized adversarial examples. Let Resnet18 [3] as example. In Resnet18, we
do not manipulate in residual block. We search adversarial examples
(Imperceptible adversarial examples, Unrecognizable adversarial examples and
adversarial patch) by controlling the feature between residual blocks. And
when defending, we add our defending strategy (e.g. Sign function) after the
first convolution which achieves the best performance. As shown in the lower
right corner of the figure, the front part of the network performs feature
extraction on pixel space. And the feature extraction is redundant. And our
defense strategy can compress information so much that in the process we lose
some information, including perturbations.

###  II-A Threat model

Recently DNNs are powerful function and have made the advanced achievements in
many domains such as Image Classify [2, 28], NLP [29, 30], Object Detection
[31, 23, 32], Semantic classification [33, 31, 34] and many more. With the
burst spread of DNNs, more and more people pay attention to the security of
DNNs. We may want to (or not want to) apply the DNNs depends on the extent to
which the adversary generates the adversarial examples.

Natural language processing (NLP) make great progress in recent years. The
model based on DNNs achieves state-of-the-art performance in various fields
such as language modelling [35, 36], syntactic parsing [37], machine
translation [38, 29] and many more. Then, the attention has transformed the
risk of the DNNs in NLP. Recent work has shown [39] it is possible to generate
adversarial examples by adding noises into texts, which fool the DNNs. This
work firstly finds adversarial examples in texts, leading an arm begins
between attack and defence in texts.

In the context of object detection, recent works find various interesting
adversarial examples [31]. For example, C. Xie et al. propose an algorithm
called Dense Adversary Generation (DAG) to generate imperceptible adversarial
examples. To decrease computing time, They utilize regional proposal network
to produce the possible targets and then sum up the loss from these targets.
Another form of adversarial examples in object detection is an adversarial
patch [23] which generate a small image to stick it on the original image.
Even these adversarial examples can attack the physical world (e.g. attacking
Yolo [32]).

In the semantic classification domain, the adversary is allowed manipulating
fewer pixels than other domain since each perturbation is responsible for at
least one-pixel segmentation [33, 31, 34]. These attacks include non-targeted
attacks and targeted attacks. However, they generate adversarial examples
which are imperceptible to humans.

This paper focuses on image classification. The model based on DNNs makes
excellent progress in image classification since Olga Russakovsky et al.
created AlexNet [40] which achieved champion in ILSVRC 2010. After that, many
excellent DNNs (such as GoogleNet [41], VGG [42], ResNet [3] etc.) are
proposed in this domain. However, when people indulge in the feast of DNNs, it
is found that the DNNs are incredibly vulnerable to adversarial examples [5].
The study of adversarial examples becomes significant in this domain.

Usually, the successful rate of adversarial attack is related to how much
distortion adversary can manipulate in original images. Szegedy et al. [5] use
L2-norm to quantify the difference between adversarial images and original
images. However, this metric is not necessarily applicable in other domain
such as NLP.

According to the knowledge of adversary, adversarial attacks can be classified
as white-box attacks and black-box attacks. In this paper, We suppose the
adversary can access the detail of DNNs, including parameters and framework.
With this strong assumption, we can construct aggressive adversarial examples
and then also utilize them to black-box attacks since previous work [17] find
the adversarial examples have a property called transferability that
perturbations crafted on an undefended model often transfer to an
adversarially trained one.

###  II-B Adversarial example

Szegedy et al. [5] firstly find the vulnerability of the DNNs. They show that
original image added small perturbation, called adversarial example, can fool
DNNs. They firstly describe searching adversarial examples as a box-
constrained optimization problem:

$\displaystyle\min$ $\displaystyle\|r\|_{2}$ (1) $\displaystyle s.t.$
$\displaystyle f(x+r)=l$ $\displaystyle x+r\in[0,1]^{m}$

$f$ means a classifier (the DNN) mapping pixel space to a discrete category
set. $x$ is the raw image and $r$ is the perturbation which is limited in
$[0,1]$. Moreover, $l$ is a targeted label. Solving this formula, we can
construct the adversarial example which the model classifies $x+r$ as $l$.

This intriguing property rises significant attention on the security of the
DNNs. Then, a significant number of works [5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15] try to generate all kinds of adversarial examples which can fool the DNNs
such as Fast Gradient Sign Method [6], Basic Iterative Method [7], Jacobian-
based Saliency Map Attack [8], DeepFool [9], CW [10], PGD [16], One Pixel
Attack [13] and so on. Goodfellow et al. [6] firstly propose based-gradient
attack, which updates along the direction of the signal function of the pixel
gradient to obtain the adversarial examples. It is a fast method since it
updates one step. Based on FGSM, Kurakin et al. [7] propose a more powerful
attack method which is multi-update to generate adversarial examples. Also
using multiple iterations, Deepfool [9] utilize a linear approximation method
to produce adversarial examples by searching the minimum distance from a clean
example to an adversarial example. Looking for more powerful and non-gradient
based attack method, Carlini et al. [10] propose various objective functions
and distance metrics to generate adversarial examples which can disable the
Distillation Defense [18]. PGD [16] is the first-order adversary which is the
most potent attack method among the first-order attack methods. Madry et al.
[16] study the adversarial robustness of DNNs and utilize projected gradient
descent to search for more aggressive adversarial examples. Their method can
significantly improve resistance to most of the adversarial attacks. However,
most of the researches search the adversarial examples follow the definition
of adversarial examples [5] which are imperceptible to humans but can fool
DNNs. Those researches focus on studying the small perturbation.

Some particular adversarial examples have been found as the development of
adversarial examples. Brown et al. [23] firstly introduce adversarial patch
attacks for image classifiers. They design a small patch than the size of the
original image and put the small patch into the original image. Those
adversarial examples are called adversarial patches those that are applied in
object detection [24, 25, 43, 44]. Moreover, many works find that adversarial
patches exist in the physical world [45, 26, 32, 46]. The adversarial patch is
not imperceptible to humans. And the perturbation usually does not confuse
humans judge. However, it can fool the DNNs.

Anh et al. [27] find that some unrecognizable for humans can be classified as
a class with high confidence by DNNs. Based evolutionary algorithms algorithm,
they proposed a new algorithm called the multi-dimensional archive of
phenotypic elites MAP-Elites [47], enable them to evolve the population
better. Sara et al. [48] show that the representation of the raw image in a
DNN can be operated to approximate those of other natural images by adding a
minor, imperceptible perturbation to the original image. They focus on the
internal layers of DNN representations. They create an adversarial example
which approximated the original image, but its internal representation appears
remarkably different from the original image. These researches are of great
significance and raise some questions about DNN representations, as well as
the vulnerability of DNN.

###  II-C Adversarial Defense

Adversarial defence [5, 16, 17, 18, 19, 20, 21, 22] makes DNNs more robust to
adversarial examples. Papernot et al. [18] propose Defensive Distillation to
defence DNN against adversarial examples. By controlling the temperature in
the distillation network, they train a DNN with hard-label and train other DNN
with soft-label. And the two networks have the same framework. However, this
defence will fail under CW [10] attack method. Adversarial training is one of
the most effective defence methods. Szegedy et al. [5] and Madry et al. [16]
use their attack method to generate adversarial examples. Then combining clean
examples and adversarial examples, they utilize them to train a DNN, which can
significant improve adversarial robustness of the DNN. However, such defence
method will fail under more powerful attack methods and is time-consuming.
Tramer et al. [17] generate adversarial examples by various adversarial
attacks and propose adversarial ensemble training which augments training data
with perturbations transferred from other models. However, recent work [49]
points out that adversarial training would cause obfuscated gradients that
lead to a false sense of security in defences against adversarial examples.
They propose three types of obfuscated gradients and design adversarial
attacks that successfully attack all defensive papers in ICLR 2018 except that
for CW [10].

###  II-D Notation

In this subsetion we will show some mathematical symbols used in this paper.
The details are shown in Table I.

TABLE I: mathematical symbols Symbol | Meaning
---|---
$\mathcal{I}$ | Pixel space
$\boldsymbol{I}$ | An example of image
$\mathcal{F}$ | Feature space
$\boldsymbol{F}$ | An example of feature of image
$\mathcal{D}$ | Dataset
$D$ | Defense strategy
$y$ | A label
$f$ | Classifier
$T$ | A function mapping $\mathcal{I}$ from to $\mathcal{F}$
$\boldsymbol{\Delta}$ | Perturbation
$P_{\boldsymbol{\Delta}}$ | Patch perturbation
$\epsilon$ | The limitation of perturbation
$\alpha,k$ | Hyperparameter

##  III Generalized Adversarial Examples

In this section, to begin, we introduce three existed types of adversarial
attack. These three types of adversarial attack include most of the current
adversarial attack methods. By analyzing the characters of these attacks, we
present a more generalized definition of adversarial examples which can
explain the existent of adversarial examples in DNNs. Then, we introduce that
the generalized adversarial examples can be degenerate currently existed
adversarial examples by adding some restricted condition.

Many works follow the definition of adversarial examples drawn by Szedegy et
al. [5]. We refer to this type of adversarial examples as Imperceptible
Adversarial Examples which is very close to the original image but can fool
the DNN. Moreover, Nguyen et al. [27] find a new type of adversarial examples
which are unrecognizable to human. We refer to it as Unrecognizable
Adversarial Examples that are hard to understand for human but is classified
by a DNN with high confidence. Another type of adversarial examples is
Adversarial Patches which are created by adding some small patches to original
images [23]. These adversarial examples have a familiar character that can
fool the DNN, but their original images are classified correctly by DNNs. The
difference between these adversarial examples is the magnitude of change in
input. The details are shown below:

  * •

Imperceptible Adversarial Examples: allows the adversary to change the whole
image limiting in bounded-norm.

  * •

Unrecognizable Adversarial Examples: allows the adversary to change the whole
image without limiting, but the adversarial examples after adding perturbation
should be unrecognizable to human.

  * •

Adversarial Patches: allows the adversary to change the image in a confined
region.

Therefore, in this paper, we want to explore the unknown type of adversarial
examples and give a definition of adversarial examples that will help us
understand adversarial examples.

Let us begin with two questions: A) If an example of $A$ class become an
example of $B$ class after adding perturbation and the DNNs misclassify it, is
it an adversarial example? A) Are adversarial examples imperceptible to
humans? For the former one, we argue that the adversary cannot change the
objective reality since all machine tasks are in the service of humanity, and
the humans understand the world according to the objective reality. In
practice, an adversarial example usually consists of a clean sample with a
slight perturbation, e.g., imperceptible adversarial sample. It is this
imperceptibility that makes it difficult for the perturbation to change the
objective reality of the original example. However, the Adversarial patch does
not take into account this imperceptibility, which may cause the sample to
become another type of sample after adding disturbance. For example, as shown
in B part if Fig. 1, the left subfigure is an English word, called ”F”. We
assume it is an objective reality (e.g. human and machine all called it as
”F”). Then we get medial subfigure by adding little pixel in the left
subfigure. These perturbation does not bother humans recognition in pixel
space, but the machine may be confused with other words in some feature space.
We consider it adversarial example if the model classifies it wrongly.
Moreover, the right subfigure also made by adding little pixel in the left
subfigure. We find that in pixel space it becomes another word which is called
”E”. It is meaningless that we call a sample as an adversarial example in this
case if the model classifies it wrongly.

For the after one, it is obvious that we can perceive the perturbation of
adversarial patches and unrecognizable adversarial examples. Furthermore, for
Imperceptible Adversarial Examples, if the limitation of perturbation is
relaxed enough, we also can perceive the small perturbation. We point out that
it is significantly different between human vision and DNNs to recognize the
images. It is a common phenomenon that the perturbation that hard to recognize
by human but may be easy to recognize for a DNN. Inversely, the perturbation
such as adversarial patches that are easy to recognize by human but maybe are
hard to recognize for a DNN. As shown in ”A” part of Fig. 1, we find that when
humans identify clean or disturbing images, the answer is always unique,
whereas DNN cannot give a definite answer. If the machine misidentified the
disturbing image, such a sample is known as an adversarial example. Besides,
if an adversarial example is imperceptible to human, the ”imperceptibility” is
defined in pixel space. Therefore, we emphasize that the human recognizes an
image in pixel space while the machine does that in a particular space which
is a machine ”vision” space (”C” part of Fig. 1). We call as Feature space.

We now draw the definition of adversarial examples which refer to it as
Generalized Adversarial Examples which include previous three types of
adversarial examples. To begin, we draw two rules about adversarial examples
as below:

  1. 1.

The adversarial example can’t change the objective reality.

  2. 2.

The adversarial example is imperceptible to machine in feature space rather
than to human vision.

According to these two rules, we assume that existing a function which can map
the original pixel space to the feature space. Then, we define generalized
adversarial examples.

###### Define 1

The generalized adversarial examples are those that do not change original
objective reality and are imperceptible to the machine in feature space and
can fool the machine.

We try to describe it formally. Let $\boldsymbol{I}$ represents the original
natural image and $y$ represents a class of $\boldsymbol{I}$. $T$ is a
function which maps the pixel space to the feature space. Moreover, $f$ maps
the feature space to the category space. Therefore, image recognition can be
described as below

$f(T(\boldsymbol{I}))=f(\boldsymbol{F})=y$

Where $T(\boldsymbol{I})=\boldsymbol{F}$. Then, the generalized adversarial
examples are described as:

  * •

If $\boldsymbol{T}$ is an identity function, Adversarial Tiny Perturbation is
the degenerate case of Generalized Adversarial Examples.

  * •

If $\boldsymbol{\Delta}$ is limited in the shape of patches, then Adversarial
Patches is the degenerate case of Generalized Adversarial Examples.

  * •

If $\boldsymbol{I+\Delta}$ is unrecognizable to human, in that case,
Generalized Adversarial Examples degenerate to Adversarial Noise Perturbation.

##  IV Generalized attack mechanism

According to the last section (Generalized adversarial examples), we get the
formal definition of Generalized Adversarial Examples. In this paper, we
mainly focus on adversarial defence. Therefore, adversarial attacks are
briefly introduced. Now we formulate the problem of searching a generalized
adversarial example for an image $\boldsymbol{I}$ as follows:

$\displaystyle\min$
$\displaystyle\|T(\boldsymbol{I})-T(\boldsymbol{I+\Delta})\|_{p}$ (2)
$\displaystyle s.t.$ $\displaystyle f(T(\boldsymbol{I+\Delta}))\not=y$

From the perspective of generalized adversarial example, we provided three
versions about searching for generalized adversarial examples. The key idea of
these attacks is searching adversarial examples by maximizing the gap between
original images and adversarial examples. The first one is tiny perturbation
version described as below:

$\displaystyle\min$
$\displaystyle\|\boldsymbol{\Delta}\|_{p}-\alpha\cdot\|T(\boldsymbol{I})-T(\boldsymbol{I+\Delta})\|_{p}$
(3) $\displaystyle s.t.$ $\displaystyle f(T(\boldsymbol{I+\Delta}))\not=y$
$\displaystyle\|\boldsymbol{\Delta}\|_{p}\leq\boldsymbol{\delta}$

where $\alpha$ is the hyperparameter and $\boldsymbol{\delta}$ is the small
value for humans perception. Eq. 3 try to find an adversarial example which is
imperceptible to human by minimizing $\boldsymbol{\delta}$ and maximize
$\|T(\boldsymbol{I})-T(\boldsymbol{I+\Delta})\|$. The pixel space for human
vision is similar to the feature space for DNNs. It is sensitive for DNNs to
the perturbation in feature space. Applying this type of attack, the
adversarial examples closed to the original image are created which are
imperceptible to humans. However, the features are extracted from the
adversarial examples by DNNs are different from that of original images. That
is why the adversarial examples can fool the DNNs.

The second one is called confidence attack, which makes DNN classifies a
natural unrecognized image for human or a natural image from other domain with
high confidence. The detailed formula is described as below:

$\displaystyle\min$
$\displaystyle\|T(\boldsymbol{I}_{in})-T(\boldsymbol{I}_{out}+\boldsymbol{\Delta})\|_{p}$
(4) $\displaystyle s.t.$ $\displaystyle f(T(\boldsymbol{I+\Delta}))\not=y$

$\boldsymbol{I}_{in}$ represents a natural image which comes from the original
domain. And $\boldsymbol{I}_{out}$ represents an image which comes from some
domain or original domain. If $\boldsymbol{I}_{out}$ is unrecognizable to
human, then we can create an adversarial example which is unrecognizable for
human but is classified by a DNN with high confidence. In that case, the
attack is similar to the work [4] that produces images that are unrecognizable
for humans, but a DNN believes to be recognizable objects with high
confidence. If $\boldsymbol{I}_{out}$ came from original domain, then this
type attack is similar to the work [1] which shows that the representation of
a raw image in a DNN can be operated to approximate those of other natural
images by adding a minor, imperceptible perturbation to the original image.
Referring to [1], $\boldsymbol{I}_{in}$ is guided image, and
$\boldsymbol{I}_{out}$ is the source image.

The third is patch version described as below:

$\displaystyle\min$
$\displaystyle-\|T(\boldsymbol{I})-T(\boldsymbol{I}+P_{\boldsymbol{\Delta}})\|_{p}$
(5) $\displaystyle s.t.$ $\displaystyle
f(T(\boldsymbol{I}+P_{\boldsymbol{\Delta}}))\not=y$ $\displaystyle
P_{\boldsymbol{\Delta}}is\ a\ small\ patch$

In Eq. 5, $P_{\boldsymbol{\Delta}}$ is a smaller patch than original image.
This attack allows adversary to manipulate pixels in confined region (e.g.
$P_{\boldsymbol{\Delta}}$) but is no limit to choose the values yielding the
max or min value. And $P_{\boldsymbol{\Delta}}$ have a specific shape such as
square, circle and so on. In practical, when optimizing the Eq. 5, we need to
know where $P_{\boldsymbol{\Delta}}$ locate in and what size that is. Usually,
the position and size of $P_{\boldsymbol{\Delta}}$ can be randomly initialized
as long as the patch does not change the objective reality of original image.
Therefore, the size of $P_{\boldsymbol{\Delta}}$ always is much smaller than
that of original image.

In Eq. 5, $P_{\boldsymbol{\Delta}}$ is a smaller patch than the original
image. This attack allows an adversary to manipulate pixels in the confined
region (e.g. $P_{\boldsymbol{\Delta}}$) but is no limit to choose the values
yielding the max or min value. Moreover, $P_{\boldsymbol{\Delta}}$ has a
specific shape such as square, circle and so on. In practical, when optimizing
the Eq. 5, we need to know where $P_{\boldsymbol{\Delta}}$ locate in and what
size that is. Usually, the position and size of $P_{\boldsymbol{\Delta}}$ can
be randomly initialized as long as the patch does not change the objective
reality of the original image. Therefore, the size of
$P_{\boldsymbol{\Delta}}$ always is much smaller than that of the original
image.

##  V Generalized defense mechanism

In this section, we introduce a defence mechanism against adversarial examples
based on generalized adversarial examples. In feature space, we define a
function $D$ which can smooth the perturbation from pixel space. Recalled that
$T$ is a function which maps pixels value to feature value and $f$ classifies
the feature to a class, we construct the defence strategy as below:

$\displaystyle\min\quad\mathop{E}_{(\boldsymbol{I},y)\backsim\mathcal{D}}[L(f\cdot
D\cdot T(\boldsymbol{I}),y)]$ (6)

where $\mathcal{D}=\\{(\boldsymbol{I}^{(i)},y^{(i)})\\}_{i=1}^{N}$ is a
dataset with $N$ samples. And $f\cdot D\cdot
T(\boldsymbol{I})=f(D(T(\boldsymbol{I})))$. $L$ is a loss function.

According to the definition of generalized adversarial examples, the
perturbations made by the adversary in pixel space are not strictly limited.
The goal of the adversary is to manipulate perturbations which can cause
significant discrepancy than original that when they transferred from pixel
space to feature space. If we can prevent the disturbance from propagating
backwards, then the model will naturally be adversarial robust. The simple
idea is to design a filter like function and then deploy it between the pixel
space and the feature space. The purpose of this function is to filter and
compress input information. Because DNNs usually have multiple convolution
kernels, the extracted features are redundant, which provides feasible
conditions for this design scheme. When the filter compresses the input
information, it loses some of the original information, including adversarial
perturbation. However, because the input information is redundant, this filter
does not have much impact on the performance of the model. Secondly, the
filter should be differentiable or segmented differentiable, which facilitates
its combination with DNN. Therefore, we propose that function $D$ have some
property from the inverse perspective of an adversary.

  1. 1.

$D$ has sufficient smoothness. It is uniform continuity or uniform continuity
in segments. This property ensures that the small perturbations in $T$ are
limited in a fixed range after they are transferring.

  2. 2.

The range of $D$ is a denumerable finite set. The critical problem of
adversarial defence is how to prevent the small perturbations slowly
amplifying in a DNN. If the interval of the domain of $D$ contains the
interval of the range, the perturbations will be compressed. This compression
naturally leads to the loss of information. However, it is this compression
effect that makes the impact of small perturbations on the results even
smaller.

Considering a single network $f(\boldsymbol{x})=\boldsymbol{W}\boldsymbol{x}$
where $\boldsymbol{W}$ is a $m\times n$ vector in feature space. $m$ is the
number of categories. And $n$ is the number of features. Let us denote
$\mathcal{I}$ as pixel space and denote $\mathcal{F}$ as feature space.
$\forall\boldsymbol{I}\in\mathcal{I}$, we have
$T(\boldsymbol{I})=\boldsymbol{F}\in\mathcal{F}$. $D$ is a defence strategy.
Then, we have the following theorem:

###### Theorem 1

Suppose $D$ satisfies the Lipschitz continuity condition, then $f\cdot D$ also
satisfies the Lipschitz continuity condition. Existing a constant $K$, we have

$\|f(D(\boldsymbol{F}+\boldsymbol{\Delta}))-f(D(\boldsymbol{F}))\|_{2}\leq
K\cdot\|\boldsymbol{\Delta}\|_{2}$

###### Proof 1

Since $D$ satisfies the Lipschitz continuity condition, then existing a
constant $c$, we have

$\|D(\boldsymbol{F+\boldsymbol{\Delta}})-D(\boldsymbol{F})\|_{2}\leq
c\|\boldsymbol{\Delta}\|$

Then,

$\begin{split}&\|f(D(\boldsymbol{F+\Delta}))-f(D(\boldsymbol{F}))\|_{2}\\\
&\leq\|\boldsymbol{W}\cdot
D(\boldsymbol{F+\Delta})+b-(\boldsymbol{W}\cdot\boldsymbol{F}+b)\|_{2}\\\
&=\|\boldsymbol{W}\cdot(D(\boldsymbol{F+\Delta})-D(\boldsymbol{F}))\|_{2}\\\
&\leq\|\boldsymbol{W}\|_{2}\cdot\|D(\boldsymbol{F+\Delta})-D(\boldsymbol{F})\|_{2}\\\
&\leq c\|\boldsymbol{W}\|_{2}\cdot\|\boldsymbol{\Delta}\|_{2}\end{split}$ (7)

Let us denote $K=c\|\boldsymbol{W}\|_{2}$, then

$(\ref{eq:theorem:1})\leq K\cdot\|\boldsymbol{\Delta}\|_{2}$

Proof done.

In Theorem 1, we give a strong supposition that $D$ satisfies the Lipschitz
continuity condition, leading $f\cdot D\cdot T$ also satisfies the Lipschitz
continuity condition. With this conclusion, the effect of perturbation can be
under-control.

Next, we will show how to design a reasonable function $D$. For satisfying the
above two property, and we propose a specific function:

$D(\boldsymbol{F})=sign(\boldsymbol{F})=\begin{cases}1,&\quad\boldsymbol{F}_{i}\geq
0\\\ -1,&\quad\boldsymbol{F}_{i}<0\end{cases}$ (8)

Since Eq. 8 is bounded, $D$ satisfies Lipschitz continuity condition. And the
range of $D$ is $\\{1,-1\\}$. In section VI, we will show that applying Eq. 8
can significantly improve the adversarial robustness of the model at the same
time, keeping accuracy on clean examples.

By observing Eq. 8, it is found that the equation is a piecewise
differentiable function whose non-differentiable point is the origin. The
defence function $D$ is too simplistic and extreme. Because the input is all
integers or all negative Numbers, the model loses all available information
through this function, resulting in network convergence failure. In the
experiment section VI, we will introduce that the deeper the network layer
where function $D$ (Eq. 8) is located, the worse the performance of the
network under clean samples. Therefore, based on equation Eq. 8, we propose a
more general defence function as following:

$D(\boldsymbol{F})=sign(\boldsymbol{F})=\begin{cases}b_{u},&\quad\boldsymbol{F}_{i}\in[a_{u},+\infty]\\\
...,&\quad\boldsymbol{F}_{i}\in[...,...]\\\
b_{l},&\quad\boldsymbol{F}_{i}\in[a_{l},-\infty]\end{cases}$ (9)

where $b_{u}$ is the upper bound, $b_{l}$ is the lower bound, and
$\\{a_{u},..,a_{l}\\}$ are the non-differentiable points.

Algorithm 1 Defense Algorithm

0: An dataset $\mathcal{D}=\\{(\boldsymbol{I}^{(i)},y^{(i)})\\}_{i=1}^{N}$; A
trained network $G=f\cdot D\cdot T=(\boldsymbol{I};\boldsymbol{\theta})$

1: Initialize hyperparameter $k$

2: Initialize epoches

3: for i=1 to epoches do

4: for $k$ steps do

5: Sample $m$ samples
$d=\\{\boldsymbol{I}^{(1)},\boldsymbol{I}^{(2)},\cdots,\boldsymbol{I}^{(m)}\\}$

6: Update the parameters by descending its stochastic gradient:
$\bigtriangledown\frac{1}{m}\sum_{i=1}^{m}{y^{(i)}\cdot
log(G(\boldsymbol{I}^{(i)}))}$

7: end for

8: end for

9: Output $\boldsymbol{\theta}$

##  VI Experiments

In this section, we use Python 3.6 and Jupyter Notebook to conduct three types
of adversarial attack method. Using toolbox Advertorch [50], we implement
several adversarial attack methods as a baseline. For imperceptible
adversarial examples, we compared our method with existed attack methods on
three datasets such as MNIST [51], CIFAR10 [52] and IMAGENET [40]. Secondly,
we use our method to generate various types of unrecognizable adversarial
examples. Finally, we test our method, which generates the adversarial
patches. In the setting in which the patch size is same, we compared our
method with existed methods.

MNIST dataset consists of 60,000 training samples and 10,000 samples, each of
which is a 28x28 pixel handwriting digital image. CIFAR10 dataset is composed
of 60,000 32x32 colour image, 50,000 for training and 10,000 for testing.
ImageNet is large colour image dataset which consists of more than 1,400,000
natural images and 1000 classes.

For any sample from MNIST, CIFAR10 and ImageNet, we make a preprocess on it,
e.g., normalization described as below:

$\boldsymbol{I}=\frac{\boldsymbol{I}-\boldsymbol{\mu}}{\boldsymbol{\sigma}}$

where $\boldsymbol{\mu}$ is the mean pixel value of the whole dataset and
$\boldsymbol{\sigma}$ is the standard deviation of that. The detail is showed
in Table II.

TABLE II: THE MEAN VALUE AND STANDARD DEVIATION IN MNIST, CIFAR10 AND IMAGENET | $\boldsymbol{\mu}$ | $\boldsymbol{\sigma}$ |  |
---|---|---|---|---
MNIST | [0.1307] | [0.3081] |  |
CIFAR10 | [0.4914, 0.4822, 0.4465] | [0.2023, 0.1994, 0.2010] |  |
ImageNet | [0.485, 0.456, 0.406] | [0.229, 0.224, 0.225] |  |

###  VI-A Imperceptible Adversarial Examples

TABLE III: A comparison of imperceptible adversarial examples with various adversarial attacks on MNIST. Method | Steps | Epsilon | Accuracy |
---|---|---|---|---
Natural | - | - | 0.9849 |
FGSM | - | 0.3 | 0.8239 |
PGD | 100 | 0.3 | 0.5838 |
SinglePixel | - | - | 0.9556 |
CW | 100 | 0.3 | 0.6052 |
LBFGS | 100 | 0.3 | 0.9972 |
L2BIA | 100 | 0.3 | 0.9758 |
Our | 100 | 0.3 | 0.5995 |
TABLE IV: A comparison of imperceptible adversarial examples with various adversarial attacks on CIFAR10. Method | Steps | Epsilon | Accuracy | Epsilon | Accuracy
---|---|---|---|---|---
Natural | - | - | 0.8892 | - | -
FGSM | - | 0.3 | 0.4221 | 0.03 | 0.743
PGD | 100 | 0.3 | 0.0049 | 0.03 | 0.4586
SinglePixel | - | - | 0.8814 | - | 0.8818
CW | 100 | 0.3 | 0.3604 | 0.03 | 0.3595
LBFGS | 100 | 0.3 | 0.9361 | 0.03 | 0.9361
L2BIA | 100 | 0.3 | 0.7739 | 0.03 | 0.8775
Our | 100 | 0.3 | 0.0303 | 0.03 | 0.4669
TABLE V: A comparison of imperceptible adversarial examples with various adversarial attacks on ImageNet. Method | Steps | Epsilon | Accuracy |
---|---|---|---|---
Natural | - | - | 0.696 |
FGSM | - | 0.03 | 0.08 |
PGD | 100 | 0.03 | 0.005 |
SinglePixel | - | - | 0.6957 |
CW | 100 | 0.03 | 0.4797 |
LBFGS | 100 | 0.03 | 0.7929 |
L2BIA | 100 | 0.03 | 0.692 |
Our | 100 | 0.03 | 0.4907 |

In this subsection, we test the first attack method (e.g. Eq. 3) on MNIST,
CIFAR10 and ImageNet. Our tiny attack method is a novel adversarial attack
method. We compare it with existed methods such as FGSM [6], PGD [16],
SinglePixel [13], CW [10], LBFGS [5] and L2BIA [7]. We utilize the LeNet [1],
ResNet18 [3] and pre-trained VGG19 [42] to test MNIST, CIFAR10 and ImageNet,
respectively. We set $\epsilon$ as 0.3 when attacking on MNIST and 0.03 on
ImageNet. Moreover, we set respectively $\alpha$ as 0.3 and 0.03 when
attacking CIFAR10. For multi-step attacks, let the number of iterations
uniformly to 100 steps.

In the best case, our attack method can drop the accuracy down to $59.95\%$
(Table III) with $\epsilon$=0.3 on MNIST. Our attack method is second only
PGD. For more challenging cases, such as imperceptible adversarial examples on
CIFAR10, our method drop the accuracy down to $3.03\%$, with $\epsilon$=0.3
(Table IV) which is second only PGD, and $46.69\%$ with $\epsilon$=0.03 (Table
IV) which is second only PGD and CW. Moreover, we test our tiny attack method
on ImageNet with $\epsilon$=0.03. The adversary drops the accuracy down to
$49.07\%$ (Table V), which is second only PGD and CW.

To validate the effectivity of our attack method, we repeat experiments by
adjusting the $\alpha$ value of Eq. 3. Fig. 3 shows the effectivity of attack
with different $\alpha$. It can be seen that the closer $\alpha$ is to zeros,
the worse the performance of attack. Moreover, we find that when locating in
[0,1], the attack achieves the best performance. As $\alpha$ gradually
increases, the success of the attack first rises rapidly and then becomes
stable.

Figure 3: A trend of attack’s power with various $\alpha$ values. We find that
closer $\alpha$ to zero, worse performance of attack.

###  VI-B Unrecognizable Adversarial Examples

In this subsection, we utilize Eq. 4 to create unrecognizable adversarial
examples to fool VGG19 neural network, which is trained with ImageNet. We
regard the images which came from ImageNet as an image from the domain
(labelled as $I_{in}$). And any images which do not belong to $I_{in}$ are out
of the domain (labelled as $I_{out}$). Our goal is creating some noise images
and regular images which are classified by DNNs as a class with high
confidence.

#### Irregular image

Firstly, we randomly generate a noise image. Then randomly choosing an image
in the domain to guide the noise to close each other in feature space. We find
that it is easy to create such a noise image. And the DNN classifies it as a
class with high confidence (Fig. 4).

#### regular image

Once again, we utilize Eq. 4 to generate unrecognizable adversarial examples.
However, those are regular images rather than noise images. To find suitable
images from out of domain, we utilize the Colored Brodatz Texture (CBT)
database and the Multiband Texture (MBT) database [53]. CBT is a coloured
version of the original 112 Brodatz grayscale texture images. And MBT is a
collection of 154 colour images. The colour of the MBT images is mainly the
result of inter-band and intra-band texture variation. Based on those images,
we search unrecognizable adversarial examples in the out domain by minimizing
the gay between the image from domain and the image from out domain in feature
space. Though those images are very different from that from ImageNet, the DNN
still mostly believes that they are from the domain (Fig. 4).

(a)  (b)  (c)

Figure 4: There are three types of adversarial examples (e.g. imperceptible
adversarial examples, unrecognizable adversarial examples and adversarial
patch). (a) Some imperceptible adversarial examples are generated from
ImageNet by the attack method of the first version (Eq. 3). (b) Some
unrecognizable adversarial examples are generated by the attack method of the
second version (Eq. 4). We utilize the noisy image and the image from the
Colored Brodatz Texture (CBT) and the Multiband Texture (MBT) dataset [53] to
produce the unrecognizable adversarial example which is classified a class
with high confidence. (c) Some adversarial patch examples are generated by the
attack method of the third version (Eq. 5).

###  VI-C Adversarial Patch

In this subsection, we try to generate adversarial patches. Adopting the
setting in [23], we set the patch size as $q\times q$ where
$q=\lfloor((Length\ of\ image)^{2}*c)^{0.5}\rfloor$. And $c$ is a constant.
Because the image size in MNIST and CIFR10 datasets are relatively small, the
generated adversarial patches are easy to modify the objective reality, so we
only generate the adversarial patches to attack VGG19 network on ImageNet. The
pixel of the patch can be manipulated freely. And the position of the patch is
randomly located. Some examples are shown in Fig. 4.

We show that our method can successfully generate adversarial patches. Since
limiting facility, we randomly choose 1,000 images from validation dataset of
ImageNet to generate their adversarial patches. Moreover, we test the accuracy
in a different setting. The details are described in Fig. 5. We find that our
attack can effectively generate adversarial patches and is better than the
Baseline [23] in most of the setting.

Figure 5: A trend of attack’s power with various sizes of patch. We find that
our attack can effectively generate adversarial patches and is better than the
Baseline [23] in most of setting.

###  VI-D Adversarial Defense

#### MNIST

We use the part of convolution layers as function $T$ and use the part of
linear layers as classifier $f$. We set epochs as 30. The learning rate is
$0.01$ at the beginning, and then half every 10 epochs. The momentum is
$0.95$. The batch size is $64$. The results of the experiment are shown in
Tabel VI. We find that our defence strategy almost invalidates all
adversaries. At the same setting, our defending method can effectively defence
gradient-based attacks (such as PGD) or non-gradient-based attacks (such as
CW).

TABLE VI: Defense on MNIST: Performance of our defending strategy against different adversaries for $\epsilon=0.3$ Method | Steps | $\epsilon$ | Natural | Adv. Training [16] | Our
---|---|---|---|---|---
Natural | - | - | 0.9849 | 0.9923 | 0.9711
FGSM | - | 0.3 | 0.8239 | 0.9795 | 0.9712
L2BIA | 100 | 0.3 | 0.9758 | 0.9503 | 0.9712
PGD | 100 | 0.3 | 0.5838 | 0.9507 | 0.9547
CW | 100 | 0.3 | 0.6052 | 0.7813 | 0.9709

#### CIFAR10

Considering a more complicated case, we use ResNet18 as our basic neural
network to train on CIFAR10. We set epochs as $30$. The learning rate is
$0.01$ at the beginning, and then half every 10 epochs. The momentum is
$0.95$. The batch size is $64$. We try to insert function $D$ into ResNet18.
The part of ResNet18 before $D$ is considered as $T$. Moreover, the part of
ResNet18 after $D$ is considered as $f$. The results of the experiment are
shown in Tabel VII. As Table VII showed, all adversaries fail to attack our
model.

TABLE VII: Defense on CIFAR10: Performance of our defending strategy against different adversaries for $\epsilon=0.03$ Method | Steps | $\epsilon$ | Natural | Adv. Training [16] | Our
---|---|---|---|---|---
Natural | - | - | 0.8892 | 0.8613 | 0.892
FGSM | - | 0.03 | 0.743 | 0.7151 | 0.8930
L2BIA | 100 | 0.03 | 0.8775 | 0.8595 | 0.8766
PGD | 100 | 0.03 | 0.4586 | 0.6943 | 0.8927
CW | 100 | 0.03 | 0.3595 | 0.0185 | 0.8903

#### ImageNet

In this part, we apply our defence method in ImageNet dataset. We adopt
ResNet50 as our model and set epochs as $90$. Then learning rate is $0.01$ at
the beginning and then is adjusted according to this formulation
$lr=0.01*0.1^{\lfloor epoch/30\rfloor}$. The momentum is $0.95$. The batch
size is $256$. The defensive function is deployed in the back of the first
convolutional layer. The detail results of the experiments are shown in Table
VIII. As a result, all adversaries fail to attack our defensive model. It is
noted that the accuracy under different attacks is almost the same with that
on clean samples. It demonstrates that our defence method can successfully
extend more realistic and complicated case.

TABLE VIII: Defense on ImageNet: Performance of our defending strategy against different adversaries for $\epsilon=0.03$ Method | Steps | $\epsilon$ | Natural | Adv. Training [16] | Our
---|---|---|---|---|---
Natural | - | - | 0.7198 | 0.4922 | 0.6453
FGSM | - | 0.03 | 0.05 | 0.0010 | 0.6443
PGD | 100 | 0.03 | 0.0003 | 0.0010 | 0.6372
CW | 100 | 0.03 | 0.0003 | 0.0009 | 0.6438

#### Analysis

We find that our defending method will decrease the accuracy of the model on
clean samples. And all adversaries fail to attack our model. Therefore, if we
can prevent the drop in the accuracy of the model on clean samples, the
adversarial robustness of the model can be further improved. To search the
best position where the function $D$ is, we do a simple test in the network
layers of LeNet and ResNet18. As shown in Fig. 6, we find that our defence
strategy is best deployed at the front of the network. The further function
$D$ is deployed, the lower the recognition rate of the model on clean samples,
and the ability of the model to resist the attack decreases accordingly.

Moreover, the recognition rate of our defence model for clean samples is very
similar to that of the model for adversarial samples. This indicates that the
adversaries have failed to attack the defence model. We present that function
$D$ acts to compress the information. After the feature passes through the
function $D$, much information is lost, including adversarial perturbation. In
the latter part of the neural network, information is extracted step by step.
If the function $D$ is placed in the back half of the neural network, a large
amount of useful information will be lost, making the performance of the model
worse.

(a) LeNet network with defense strategy (b) Resnet18 network with defense
strategy

Figure 6: The effect of setting location of function D on network performance.
(a) Lenet have five layers include two convolutions and three connective
layers. (b) ResNet18 have a convolution layer, a fully connective layer and
four blocks.

Moreover, we analyze why function $D$ can effectively play a defensive role by
visualizing the changes in the input found after convolution. We use the
trained ResNet50 network to visualize the input, the output of the first layer
of convolution, and the output after function $D$. Function $D$ is Eq. 8. We
compare the degree of contamination of inputs under different disturbances. As
Fig. 7 shown, when function $D$ filters the information, it filters the
adversarial perturbations to some extent. When $\epsilon\leq 0.06$, function
$D$ filters out adversarial perturbations almost entirely (Columns 2 and 3).
In the case without function $D$, the input is heavily contaminated (column 1
and column 4).

Figure 7: We use the trained ResNet50 network to visualize the input, the
output of the first layer of convolution and the output after function $D$.
Function $D$ is Eq. 8. We compare the degree of contamination of inputs under
different disturbances.

##  VII Conclusion

We show that, according to our definition of adversarial examples, we can
generate imperceptible adversarial examples, unrecognizable adversarial
examples and adversarial patches and construct defending strategy. Our attack
methods can effectively generate adversarial examples whose performances are
better than the most attack methods (except for PGD and CW in some setting).
Furthermore, our defending strategy achieve state-of-the-art performance. As
far as we know, our defence is the best performance than existed methods such
as adversarial training. Since we do not use adversarial training, our defence
is less time-consuming than adversarial training.

In this paper, we define generalized adversarial examples to include three
types of adversarial examples. Furthermore, our attack methods are focus on a
single hidden layer. Starting from our definition, we propose three attack
methods and one defence strategy. We present that we can construct more
powerful attack methods by manipulating multi-hidden layers or adaptively
manipulating multi-hidden layers. That is what we want to improve in the
future. Our defence strategy achieves excellent performance. However, even if
our defence method can defend against the attack, the improper setting will
lead to the model’s accuracy decreased in the clean sample. Therefore, there
is room for improvement in our defence in the future.

## Acknowledgment

This work was supported in part by Key Project of Natural Science Foundation
of China (Grant 61732011), in part by the National Natural Science Foundation
of China (Grants 61976141, 61772344 and 61732011), in part by the Natural
Science Foundation of SZU (827-000230), and in part by the Interdisciplinary
Innovation Team of Shenzhen University.

## References

  * [1] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [3] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [4] C. Olah, “Understanding lstm networks,” 2015.
  * [5] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2014. [Online]. Available: http://arxiv.org/abs/1312.6199
  * [6] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available: http://arxiv.org/abs/1412.6572
  * [7] A. Kurakin, I. J. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” in _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings_. OpenReview.net, 2017\. [Online]. Available: https://openreview.net/forum?id=HJGU3Rodl
  * [8] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “The limitations of deep learning in adversarial settings,” in _2016 IEEE European symposium on security and privacy (EuroS &P)_. IEEE, 2016, pp. 372–387.
  * [9] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2574–2582.
  * [10] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _2017 ieee symposium on security and privacy (sp)_. IEEE, 2017, pp. 39–57.
  * [11] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh, “Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” in _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , 2017, pp. 15–26.
  * [12] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Universal adversarial perturbations,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1765–1773.
  * [13] J. Su, D. V. Vargas, and K. Sakurai, “One pixel attack for fooling deep neural networks,” _IEEE Transactions on Evolutionary Computation_ , vol. 23, no. 5, pp. 828–841, 2019.
  * [14] Z. Zhao, D. Dua, and S. Singh, “Generating natural adversarial examples,” _arXiv preprint arXiv:1710.11342_ , 2017.
  * [15] P. Tabacof, J. Tavares, and E. Valle, “Adversarial images for variational autoencoders,” _arXiv preprint arXiv:1612.00155_ , 2016.
  * [16] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018. [Online]. Available: https://openreview.net/forum?id=rJzIBfZAb
  * [17] F. Tramèr, A. Kurakin, N. Papernot, I. J. Goodfellow, D. Boneh, and P. D. McDaniel, “Ensemble adversarial training: Attacks and defenses,” in _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018. [Online]. Available: https://openreview.net/forum?id=rkZvSe-RZ
  * [18] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, “Distillation as a defense to adversarial perturbations against deep neural networks,” in _2016 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2016, pp. 582–597.
  * [19] R. Huang, B. Xu, D. Schuurmans, and C. Szepesvári, “Learning with a strong adversary,” _arXiv preprint arXiv:1511.03034_ , 2015.
  * [20] M. Cheng, Q. Lei, P.-Y. Chen, I. Dhillon, and C.-J. Hsieh, “Cat: Customized adversarial training for improved robustness,” _arXiv preprint arXiv:2002.06789_ , 2020.
  * [21] C. Song, K. He, L. Wang, and J. E. Hopcroft, “Improving the generalization of adversarial training with domain adaptation,” in _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019. [Online]. Available: https://openreview.net/forum?id=SyfIfnC5Ym
  * [22] H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan, “Theoretically principled trade-off between robustness and accuracy,” _arXiv preprint arXiv:1901.08573_ , 2019.
  * [23] T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer, “Adversarial patch,” _CoRR_ , vol. abs/1712.09665, 2017. [Online]. Available: http://arxiv.org/abs/1712.09665
  * [24] X. Liu, H. Yang, Z. Liu, L. Song, Y. Chen, and H. Li, “DPATCH: an adversarial patch attack on object detectors,” in _Workshop on Artificial Intelligence Safety 2019 co-located with the Thirty-Third AAAI Conference on Artificial Intelligence 2019 (AAAI-19), Honolulu, Hawaii, January 27, 2019_ , ser. CEUR Workshop Proceedings, H. Espinoza, S. Ó. hÉigeartaigh, X. Huang, J. Hernández-Orallo, and M. Castillo-Effen, Eds., vol. 2301. CEUR-WS.org, 2019. [Online]. Available: http://ceur-ws.org/Vol-2301/paper\\_5.pdf
  * [25] D. Karmon, D. Zoran, and Y. Goldberg, “Lavan: Localized and visible adversarial noise,” in _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , ser. Proceedings of Machine Learning Research, J. G. Dy and A. Krause, Eds., vol. 80. PMLR, 2018, pp. 2512–2520. [Online]. Available: http://proceedings.mlr.press/v80/karmon18a.html
  * [26] M. Lee and Z. Kolter, “On physical adversarial patches for object detection,” _arXiv preprint arXiv:1906.11897_ , 2019.
  * [27] A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 427–436.
  * [28] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _Advances in neural information processing systems_ , 2015, pp. 91–99.
  * [29] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _Advances in neural information processing systems_ , 2014, pp. 3104–3112.
  * [30] H. Xu, M. Dong, D. Zhu, A. Kotov, A. I. Carcone, and S. Naar-King, “Text classification with topic-based word embedding and convolutional neural networks,” in _Proceedings of the 7th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics_ , 2016, pp. 88–97.
  * [31] C. Xie, J. Wang, Z. Zhang, Y. Zhou, L. Xie, and A. Yuille, “Adversarial examples for semantic segmentation and object detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1369–1378.
  * [32] S. Thys, W. Van Ranst, and T. Goedemé, “Fooling automated surveillance cameras: adversarial patches to attack person detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–0.
  * [33] V. Fischer, M. C. Kumar, J. H. Metzen, and T. Brox, “Adversarial examples for semantic image segmentation,” in _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings_. OpenReview.net, 2017. [Online]. Available: https://openreview.net/forum?id=S1SED1MYe
  * [34] ——, “Adversarial examples for semantic image segmentation,” _arXiv preprint arXiv:1703.01101_ , 2017.
  * [35] T. Mikolov, S. Kombrink, L. Burget, J. Černockỳ, and S. Khudanpur, “Extensions of recurrent neural network language model,” in _2011 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2011, pp. 5528–5531.
  * [36] R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu, “Exploring the limits of language modeling,” _arXiv preprint arXiv:1602.02410_ , 2016.
  * [37] E. Kiperwasser and Y. Goldberg, “Simple and accurate dependency parsing using bidirectional lstm feature representations,” _Transactions of the Association for Computational Linguistics_ , vol. 4, pp. 313–327, 2016.
  * [38] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available: http://arxiv.org/abs/1409.0473
  * [39] N. Papernot, P. McDaniel, A. Swami, and R. Harang, “Crafting adversarial input sequences for recurrent neural networks,” in _MILCOM 2016-2016 IEEE Military Communications Conference_. IEEE, 2016, pp. 49–54.
  * [40] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [41] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [42] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available: http://arxiv.org/abs/1409.1556
  * [43] M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter, “Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition,” in _Proceedings of the 2016 acm sigsac conference on computer and communications security_ , 2016, pp. 1528–1540.
  * [44] K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song, “Robust physical-world attacks on deep learning visual classification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1625–1634.
  * [45] A. Liu, X. Liu, J. Fan, Y. Ma, A. Zhang, H. Xie, and D. Tao, “Perceptual-sensitive gan for generating adversarial patches,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 1028–1035.
  * [46] S.-T. Chen, C. Cornelius, J. Martin, and D. H. P. Chau, “Shapeshifter: Robust physical adversarial attack on faster r-cnn object detector,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2018, pp. 52–68.
  * [47] A. Cully, J. Clune, and J.-B. Mouret, “Robots that can adapt like natural animals,” _arXiv preprint arXiv:1407.3501_ , vol. 2, 2014.
  * [48] S. Sabour, Y. Cao, F. Faghri, and D. J. Fleet, “Adversarial manipulation of deep representations,” in _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2016. [Online]. Available: http://arxiv.org/abs/1511.05122
  * [49] A. Athalye, N. Carlini, and D. A. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” in _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , ser. Proceedings of Machine Learning Research, J. G. Dy and A. Krause, Eds., vol. 80. PMLR, 2018, pp. 274–283. [Online]. Available: http://proceedings.mlr.press/v80/athalye18a.html
  * [50] G. W. Ding, L. Wang, and X. Jin, “AdverTorch v0.1: An adversarial robustness toolbox based on pytorch,” _arXiv preprint arXiv:1902.07623_ , 2019.
  * [51] L. Deng, “The mnist database of handwritten digit images for machine learning research [best of the web],” _IEEE Signal Processing Magazine_ , vol. 29, no. 6, pp. 141–142, 2012.
  * [52] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [53] S. Abdelmounaime and H. Dong-Chen, “New brodatz-based image databases for grayscale color and multiband texture analysis,” _ISRN Machine Vision_ , vol. 2013, 2013.