# Adversarial Medical Image with Hierarchical Feature Hiding

Qingsong Yao  Zecheng He  Yuexiang Li  Yi Lin  Kai Ma  Yefeng Zheng  Fellow,
IEEE
and S. Kevin Zhou  Fellow, IEEE Zhou is with School of Biomedical Engineering
& Suzhou Institute for Advanced Research, University of Science and Technology
of China and Institute of Computing Technology, Chinese Academy of Sciences.
Zhou is the corresponding author. Email<EMAIL_ADDRESS>is with
Institute of Computing Technology, Chinese Academy of Science. Email:
<EMAIL_ADDRESS>Ma and Zheng are with Jarvis Research
Center, Tencent YouTu Lab, Shenzhen, P.R. China. Emails: {vicyxli, kylekma,
<EMAIL_ADDRESS>is also with Medical AI ReSearch (MARS) Group,
Guangxi Key Laboratory for Genomic and Personalized Medicine, Guangxi Medical
University, Nanning, P.R. China.Lin is with the Department of Computer Science
and Engineering, The Hong Kong University of Science and Technology.He is with
Meta Reality Labs. Email<EMAIL_ADDRESS>
###### Abstract

Deep learning based methods for medical images can be easily compromised by
adversarial examples (AEs), posing a great security flaw in clinical decision-
making. It has been discovered that conventional adversarial attacks like PGD
which optimize the classification logits, are easy to distinguish in the
feature space, resulting in accurate reactive defenses. To better understand
this phenomenon and reassess the reliability of the reactive defenses for
medical AEs, we thoroughly investigate the characteristic of conventional
medical AEs. Specifically, we first theoretically prove that conventional
adversarial attacks change the outputs by continuously optimizing vulnerable
features in a fixed direction, thereby leading to outlier representations in
the feature space. Then, a stress test is conducted to reveal the
vulnerability of medical images, by comparing with natural images.
Interestingly, this vulnerability is a double-edged sword, which can be
exploited to hide AEs. We then propose a simple-yet-effective hierarchical
feature constraint (HFC), a novel add-on to conventional white-box attacks,
which assists to hide the adversarial feature in the target feature
distribution. The proposed method is evaluated on three medical datasets, both
2D and 3D, with different modalities. The experimental results demonstrate the
superiority of HFC, _i.e.,_ it bypasses an array of state-of-the-art
adversarial medical AE detectors more efficiently than competing adaptive
attacks111Our code is available at
https://github.com/qsyao/Hierarchical_Feature_Constraint., which reveals the
deficiencies of medical reactive defense and allows to develop more robust
defenses in future.

Security in Machine Learning, adversarial attacks and defense

##  1 Introduction

Deep neural networks (DNNs) are vulnerable to adversarial examples (AEs) [1],
which are maliciously computed by adding human-imperceptible perturbations to
clean images. Those AEs can compromise a network to generate the attacker-
desired incorrect predictions [2]. Accordingly, the adversarial attack in
medical imaging analysis [3] is disastrous. For example, the attacker can
manipulate the diagnosis of patients in order to raise malicious competition
between AI software vendors, threaten the patient’s health [4], and even cause
public distrust of the DNN-based diagnosis system. More disturbingly, it has
been investigated that medical DNNs [5, 6, 7], including organ segmentation
[8], disease diagnosis [9, 10, 11], and landmark detection [12], are more
vulnerable to adversarial attacks than those for natural images.

Nevertheless, conventional medical AEs like PGD [13] which optimize the
classification logits can be easily detected in the feature space [14]. In
Fig. 1, we plot a 2D t-SNE [15] to illustrate the differences between
adversarial and clean features from the penultimate layer of a well-trained
pneumonia classifier [16]. It can be observed that adversarial attacks move
the feature from the original clean distribution to extreme outlier positions.
Therefore, reactive defenders [17] can easily make use of this characteristic
for the identification of AEs [18].

Figure 1: We craft adversarial examples by the state-of-the-art basic
iterative method (BIM) [19] under a small constraint $L_{\infty}=1$ to
manipulate the medical diagnosis result. Then, we visualize the penultimate
layer’s deep representations of the clean and adversarial examples by 2D
t-SNE. Obviously, the adversarial feature lies in an outlier position that can
be easily detected by adversarial detectors.

Since the development process of adversarial attack and defense approaches is
an alternate progress, stronger attacks inspire more robust defenses and vice
versa, which will both lead to the final robust solution in the future. In
this paper, we investigate the underlying reason for this phenomenon and
reassess the robustness of the reactive defenses. We raise two important
questions. The first one is: Why are medical AEs easier to be detected,
compared to AEs of natural images? To further reveal the underlying mechanism,
we conduct both theoretical and empirical analyses. First, we theoretically
prove that the adversarial features are optimized in a consistent direction
towards outlier points, where the clean features rarely reside. This applies
to both natural and medical images. Then, we conduct a stress test, which aims
to change the features by adversarial attacks, and find that medical features
are more vulnerable than natural ones in this test. Hence, the adversarial
features of medical images are pushed more drastically to a consistent
direction, and become outliers to a more severe extent than those of natural
images, which are easier to distinguish.

The second question is: Is it possible to hide a medical AE from being
detected by existing defenders in the feature space? A representative adaptive
attack chooses a guiding example and pushes the feature of AE to be close to
that of the guiding example [20]. However, there is a limitation when directly
applying representative adaptive attacks to medical AEs: different medical
images contain different backgrounds and lesion areas, which makes it
difficult to manipulate the AE features to be the same to the guiding one in
all layers within the perturbation budget. To hide the adversarial feature
with a low detectability, we propose a simple-yet-effective hierarchical
feature constraint (HFC) as a novel add-on term, which can be plugged into any
existing white-box attack. Our HFC first models the clean feature
distributions with a Gaussian mixture model for each layer, then encourages
adversarial features to move towards the high-density area by maximizing the
log-likelihood.

Extensive experiments are conducted on different datasets, including two
public datasets on 2D images and one private dataset on 3D volumes, with
different backbones and perturbation budgets to validate the effectiveness of
our HFC. The experimental results show that the proposed HFC helps medical AEs
compromise the state-of-the-art (SOTA) adversarial detectors, and keeps the
perturbations small and unrecognizable to humans, which reveals the
deficiencies of these medical reactive defenses that only detect medical AEs
using OOD metrics. Moreover, we demonstrate that HFC greatly outperforms other
adaptive attacks [20, 21, 22] on manipulating adversarial features. Our
experimental results support that the greater vulnerability of medical
features allows the attacker more room for malicious manipulation. we believe
that our work inspires future defenses that focus on improving the robustness
of vulnerable medical features to become more robust.

Overall, we highlight the following contributions:

  * •

We theoretically and experimentally investigate the underlying reason to
explain why medical AEs are more easily detected in feature space, compared
with natural AEs.

  * •

We propose the hierarchical feature constraint (HFC), a novel plug-in that can
be applied to existing white-box attacks to avoid being spotted.

  * •

Extensive experimental results validate the effectiveness of our HFC to help
conventional adversarial attacks bypass an array of state-of-the-art
adversarial detectors within a small perturbation budget, which exposes the
limitations of reactive medical defense and offers insights into future
stronger defenses.

A preliminary version [23] of this work was previously published as a
conference paper.222The reviewers’ comments can be found in
https://miccai2021.org/openaccess/paperlinks/2021/09/01/017-Paper1101.html.
This paper substantially extends the preliminary version with the following
improvements:

  * •

We provide a detailed proof of the theorem about feature vulnerability for the
binary class setting. Furthermore, we expand the theorem to multi-class and
provide both empirical and theoretical analyses for the out-of-distribution
adversarial feature in this new multi-class setting.

  * •

We additionally evaluate the performance of our HFC on a multi-class disease
classification dataset consisting of 3D medical volumes, as well as with
different baseline models. In addition, we find that 3D medical volumes are
more vulnerable than 2D medical images.

  * •

We provide more detailed discussion of experimental results, hyper-parameter
analysis, and compare the performances of HFC on medical and natural image
datasets, and reveal the importance of the robustness of medical deep
representations.

  * •

We further evaluate the potential of our HFC term to detect the real outlier
signals, by utilizing the HFC term to serve as the metric of the anomaly
score, which can detect the abnormal features caused by the synthesized
anomaly signals.

##  2 Related Work

Given a clean input image $x$ with its ground truth label
$y\in[1,2,\ldots,K]$, a DNN classifier $h$ with pretrained parameters $\theta$
predicts the class label $y^{\prime}$ of the input $x$ via:

$\displaystyle y^{\prime}$
$\displaystyle=\mathop{\arg\max}\limits_{k}~{}p_{\theta}(k|x)\equiv\frac{\mathop{\exp}(l_{k}(x,\theta))}{\sum_{j=1}^{K}\mathop{\exp}(l_{j}(x,\theta))},$
(1)

where the logit output with respect to class $k$, $l_{k}(x,\theta)$, is given
as $l_{k}(x,\theta)=\sum_{n=1}^{N}{w_{nk}*z_{n}(x,\theta)}+b_{k}$, in which
$z_{n}(x,\theta)$ is the $n^{th}$ activation of the penultimate layer with $N$
dimensions; $b_{k}$ and $w_{nk}$ are the bias and the weights from the final
dense layer, respectively; and the $p_{\theta}(k|x)$ is the probability of $x$
belonging to class $k$.

A common way of crafting an AE is to manipulate the classifier’s outputs by
minimizing333In this work, we focus on targeted adversarial attack that aims
to manipulate the output class for an input to a desired label, rather than
just an incorrect label. the classification error between the outputs and the
target class $c$, while keeping the AE $x_{adv}$ within a small
$\epsilon$-ball of the $L_{p}$-norm [13] centered at the original clean input
$x$, i.e., $\|x_{adv}-x\|_{p}\leq\epsilon$, where $\epsilon$ is perturbation
budget.

###  2.1 Adversarial attacks

A wide range of gradient-based or optimization-based attacks have been
proposed to generate AEs under different types of norms. DeepFool [24] is an
$L_{2}$-norm attack method which performs a minimum amount of perturbations by
moving the attacked input sample to its closest decision boundary. Another
effective $L_{2}$-norm attack proposed by Carlini and Wagner (CW attack) [25]
takes a Lagrangian form and uses Adam [26] for optimization.

In this work, we focus on SOTA $L_{\infty}$ adversarial attacks, which are
mostly used according to its consistency with respect to human perception
[13]. The conventional $L_{\infty}$ approaches can be categorized into three
categories. The first category is a one-step gradient-based approach, such as
the fast gradient sign method (FGSM) [27], which generates an adversarial
example $x^{*}$ by minimizing the loss $J(h_{\theta}(x^{*}),c)$. $J$ is often
chosen as the cross-entropy loss and $\epsilon$ is the $L_{\infty}$ norm
bound:

$\displaystyle
x^{*}=x-\epsilon\cdot\mathrm{sign}(\nabla_{x}J(h_{\theta}(x^{*}),c)),$ (2)

The second category is an iterative method. The basic iterative method (BIM)
[19] is an iterative version of FGSM, which iteratively updates perturbations
with a smaller step size $\alpha$ and keep the perturbation in the
$L_{\infty}$ norm bound by the projection function $\Pi(\cdot)$:

$\displaystyle
x_{t+1}^{*}=\Pi_{\epsilon}(x_{t}^{*}-\alpha\cdot\mathrm{sign}(\nabla_{x}J(h_{\theta}(x_{t}^{*}),c))),~{}x_{0}^{*}=x,$
(3)

Different from BIM, another iterative method named projected gradient descent
(PGD) [13] uses a random start $x_{0}=x+U^{d}(-\epsilon,\epsilon)$, where
$U^{d}(-\epsilon,\epsilon)$ is the uniform noise between $-\epsilon$ and
$\epsilon$, and perturbs the input iteratively by Eq. (3). Furthermore, the
momentum iterative method (MIM) [2] is proposed to improve the transferability
by integrating the momentum term into the iterative process. Translation-
Invariant Attacks (TIM) [28] boosts BIM by optimizing a perturbation over an
ensemble of translated images.

The last category is optimization-based method, among which one of the
representative methods is the Carlini and Wanger (CW) attack [25]. According
to [13], the $L_{\infty}$ version of CW attack can be solved by the PGD
algorithm using the following loss function:

$\displaystyle\hat{J}=\max(l_{c}(x_{t}^{*},\theta)-l_{{y_{max}\neq
c}}(x_{t}^{*},\theta),-\kappa).$ (4)

where $l_{y_{max}\neq c}$ is the largest logit of the remaining classes except
for class $c$; $l_{c}$ is the logit with respect to the target class; and
$\kappa$ is the parameter controlling the confidence.444We set $\kappa$ to the
average difference between the largest logit and the penultimate logits for
each dataset. $\hat{J}$ can better compromise the classifier by making the
second largest logit $l_{{y_{max}\neq c}}$ smaller than $l_{c}-\kappa$ [25].
It is worth noting that since our HFC is an add-on, it works with any of the
aforementioned attacking method.

###  2.2 Adversarial defenses

Plenty of proactive defense approaches have been proposed to clear the
adversarial perturbation, such as network distillation [29], feature squeezing
[30], input smoothing transformation (_e.g.,_ autoencoder-based denoising
[31], JPEG compression [32] and regularization [33]), Parseval network [34],
gradient masking [35], randomization [36, 37], radial basis mapping kernel
[38], and non-local context encoder [39]. According to [40], the PGD-based
adversarial (re)training [27, 41, 13] achieves the strongest robustness by
augmenting the training set with adversarial examples, but consumes accuracy
and training time. However, these defenses can be bypassed either completely
or partially by adaptive attacks [21, 22, 42].

Another line of research is reactive defense, aiming at detecting AEs with
high accuracy [43, 44, 45]. In particular, several emerging works shed light
on the intrinsic characteristic in the high dimensional feature subspace [45,
46]. Some of them use learning-based methods (_e.g.,_ DNN [47], RBF-SVM [48])
to train a decision boundary between adversarial and clean distributions in
the feature space. Others use k-nearest neighbors (kNN) [49, 50, 51] based
methods, which make prediction according to logits (or classes) of the kNNs in
the feature space. Furthermore, anomaly-detection based methods are suitable
for detecting AEs too. Feinman et al. [52] and Li et al. [14] model the normal
distribution with kernel density estimation (KDE) and multivariate Gaussian
model (MGM), respectively. Ma et al. [17] characterize the dimensional
properties of the adversarial subspaces by local intrinsic dimensionality
(LID). Lee et al. [18] measure the degree of the outlier by a Mahalanobis
distance (MAHA) based confidence score.

Regarding the detectability of medical AEs, Ma et al. [11] evaluate that
medical AEs are much easier to detect than natural images (with 100%
accuracy). A similar conclusion comes from [14], which motivates us to explore
the reason behind this phenomenon and evaluate the robustness of those
detectors.

Figure 2: The similarity between the value changes (under adversarial attack)
from the penultimate layer and the difference between $w_{i1}$ and $w_{i0}$.
The activation value of the component with a greater difference is increased
higher after the attack.  Figure 3: The similarity of the changes of
activation values from the penultimate layer between different attacks and the
different iterations from the basic iterative method (BIM) for both binary (on
Fundoscopy [53] dataset) and multi-class (on CIFAR-10 [54]) classification
networks. Here, we choose the components ‘Consistent_0’ in the penultimate
layer for multi-class setting, whose gradients keep negative in each iteration
of BIM. The adversarial attacks are pushed to similar directions between
different attacks and iterations.

##  3 Are Medical AEs Easy to Detect?

###  3.1 Consistency of gradient direction

To investigate how the vulnerability of feature representations can be
exploited by adversarial attacks, we focus on the object function of the final
logit output $l_{k}(x,\theta)$ and its corresponding gradient. In each
iteration of the $L_{\infty}$ approaches introduced in Sec. 2.1, $J$ and
$\hat{J}$ simultaneously increase the logit of the target class and decrease
the other logits. Therefore, the gradients are towards the similar directions
in different iterations of various attacks, which are back-propagated to all
of the layers in the diagnosis network according to the chain rule. In this
section, we provide empirical and theoretical analyses for both binary and
multi-class settings, and discuss the implications in detail.

####  3.1.1 Binary classification

Theorem I. Consider a binary disease diagnosis network and its representations
from the penultimate layer, the directions of the corresponding gradients are
fixed during each iteration under adversarial attack.

Proof: For simplicity, we ignore the bias parameters in the penultimate layer,
which does not affect the conclusion. Formally, the cross-entropy loss is
defined as:

$J_{CE}=-\sum_{k}y_{k}log(p_{k}),$ (5)

where $p_{k}$ is computed according to Eq. (1).

According to the chain rule, we compute the partial derivative of $J$ with
respect to $l_{i}$:

$\nabla_{l_{k}}J(h_{\theta}(x))=p_{k}-y_{k}.$ (6)

The binary diagnosis network has two classes $k\in[0,1]$. Hence,

$\begin{split}\nabla_{z_{i}}J(h_{\theta}(x))&=\nabla_{l_{0}}J\times\nabla_{z_{i}}l_{0}+\nabla_{l_{1}}J\times\nabla_{z_{i}}l_{1}\\\
&=(p_{0}-y_{0})w_{i0}+(p_{1}-y_{1})w_{i1}.\end{split}$ (7)

Without loss of generality, suppose that the goal of targeted adversarial
attack is to invert the prediction from 0 to 1, i.e., $y_{0}=0,y_{1}=1$. So we
can derive the partial derivative on $z_{i}$:

$\begin{split}\nabla_{z_{i}}J(h_{\theta}(x),y_{1})&=p_{0}w_{i0}+(p_{1}-1)w_{i1}\\\
&=(1-p_{1})(w_{i0}-w_{i1}),\end{split}$ (8)

where $1-p_{1}>0$ and $w_{i0}-w_{i1}$ is constant, so the partial derivative
on the feature of penultimate layer $z_{i}$ keeps in the same direction.

Implication. Accordingly, guided by the gradient, the component $z_{i}$ with
the bigger difference between $w_{i1}$ and $w_{i0}$ increases more drastically
under adversarial attack. We plot the similarity between $w_{i0}-w_{i1}$ and
the value changes in Fig. 2. And, the similarities of the value changes of the
features in the penultimate layer among different adversarial approaches and
different iterations are calculated and presented in Fig. 3. As shown, the BIM
attack pushes the features toward similar directions in each iteration.
Furthermore, various adversarial attacks push the feature towards a similar
direction by increasing the targeted logit and decreasing the left ones, which
empirically validates our hypothesis: the features are pushed toward a similar
direction during different iterations of different attacks. As a consequence,
the adversarial features are pushed to be the outliers and easy to be detected
by the reactive defenses.

####  3.1.2 Multi-class classification

Theorem II. Similar to binary classification, some of the activation values
are pushed to outlier, which are guided by the gradients toward similar
direction in each iteration. Differently, the derivative result shows that
only the channels ($i$) with the biggest weight $w_{ic}$ (greater than other
weights $w_{ik},k\neq c$) have the negative gradients all the time. We mark
these channels as ‘Consistent_c’.

Proof: The goal is to make the prediction classified to a particular erroneous
class $c$. The partial derivative of $J$ with respect to the $i$-th activation
of the penultimate layer is computed as:

$\nabla_{z_{i}}J(h_{\theta}(x))=\sum_{k=0}^{K}\nabla_{l_{k}}J\times\nabla_{z_{i}}l_{k}.$
(9)

Using Eq. (6), we have:

$\begin{split}\nabla_{z_{i}}J(h_{\theta}(x))=\sum_{k=0}^{K}(p_{k}-y_{k})w_{ik}.\end{split}$
(10)

For targeted attack, we know that:

$y_{k}=\begin{cases}1,&\text{if $k=c$}\\\ 0,&\text{if $k\neq c$}.\\\
\end{cases}$ (11)

Hence, Eq. (10) can be rewritten as:

$\begin{split}\nabla_{z_{i}}J(h_{\theta}(x),y_{c})&=\sum_{k\neq
c}^{K}p_{k}w_{ik}+(p_{c}-1)w_{ic}\\\ &=\sum_{k\neq
c}^{K}p_{k}(w_{ik}-w_{ic}),\end{split}$ (12)

where $p_{k}\geq 0$ and $(w_{ik}-w_{ic})$ is constant. Different from the
binary setting where all of gradients $\nabla_{z_{i}}J(h_{\theta}(x),y_{c})$
of the components $z_{i}$ can keep positive or negative in each iteration, we
focus on the component $z_{i}$ of the channels ‘Consistent_c’ whose $w_{ic}$
is no smaller than the other weights $w_{ik}$:

$\forall k\in K,w_{ik}-w_{i\mathrm{c}}\leq 0,$ (13)

where gradient $\nabla_{z_{i}}J(h_{\theta}(x),y_{c})$ keeps negative all the
time.

Implication. Similar to binary classification, we calculate the changes of the
activation values from the components ‘Consistent_0’ in the penultimate layer.
The results are shown in Fig. 3, which validate Theorem 2: the features of
these ‘Consistent_c’ components are pushed toward similar direction during
each iteration of the adversarial attack. Accordingly, these features of
components ‘Consistent_c’ are optimized toward similar direction into abnormal
regions and can be easily distinguished by adversarial detectors.

Figure 4: Comparison of the vulnerability between medical images (Fundoscopy
[53] and Chest X-ray [55]) and natural images (CIFAR-10 [54] and ImageNet
[56]). We calculate the changed ratios of the activation value from different
blocks in ResNet-50 before and after the adversarial attacks. The stress test
uses BIM [19] with perturbations under the constraint $L_{\infty}=1/256$, and
optimizes $J^{*}$ for 10 iterations.

###  3.2 Vulnerability of representations

We perform a stress test to empirically evaluate the robustness of the
features of natural and medical images. In practice, we use adversarial attack
to manipulate the deep representations as much as possible. In the
experiments, we then use BIM [19] to generate adversarial examples, which
decrease ($\downarrow$) and increase ($\uparrow$) the features, by optimizing
the new loss function $J_{\downarrow}^{*}=\mathrm{mean}(f^{l}(x))$ and
$J_{\uparrow}^{*}=-\mathrm{mean}(f^{l}(x))$ in each iteration, respectively,
where $f^{l}(x)$ is the feature from the $l^{th}$ activation layer. The stress
test is executed on the medical image dataset (Fundoscopy [53] and Chest X-ray
[55]) and natural image dataset (CIFAR-10 [54] and ImageNet [56]). The
experimental results shown in Fig. 4 demonstrate that the features of the
medical images can be altered more drastically, _i.e.,_ the medical image
representations are much more vulnerable than natural images.

Table 1: The point-wise results of the proposed HFC. The metric scores on the left of $\rightarrow$ are the accuracy (%) of the adversarial detectors, trained on the AEs generated by the corresponding adversarial attacks with the constraint $L_{\infty}=1/256$. The metrics on the right of $\rightarrow$ are the adversarial detection accuracy under our attack ($\downarrow$), which satisfies the same constraint. All of the proposed attacks are under constraint $L_{\infty}=1/256$ and evaluated on ResNet-50 (2D images) or ResNet-18 (3D volumes). Adv. Acc is the successful rate ($\uparrow$) of HFC to manipulate the prediction of disease diagnosis network.  Fundoscopy | MIM (Adv. Acc=99.5%) | TIM (Adv. Acc=95.9%) | PGD (Adv. Acc=99.5%) | CW (Adv. Acc=99.5%)
---|---|---|---|---
AUC | TPR@90 | AUC | TPR@90 | AUC | TPR@90 | AUC | TPR@90
KD [52] | 98.8 | $\rightarrow$ | 72.0 | 96.3 | $\rightarrow$ | 10.0 | 99.3 | $\rightarrow$ | 69.9 | 98.2 | $\rightarrow$ | 18.2 | 99.4 | $\rightarrow$ | 73.4 | 98.6 | $\rightarrow$ | 13.2 | 99.5 | $\rightarrow$ | 74.7 | 99.1 | $\rightarrow$ | 19.6
MAHA [18] | 100 | $\rightarrow$ |  7.8 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ |  8.5 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ |  4.2 | 100 | $\rightarrow$ |  0.0 | 99.8 | $\rightarrow$ | 33.0 | 99.5 | $\rightarrow$ |  0.0
LID [17] | 98.8 | $\rightarrow$ | 67.1 | 99.1 | $\rightarrow$ | 31.5 | 98.0 | $\rightarrow$ | 59.4 | 93.6 | $\rightarrow$ | 17.2 | 99.6 | $\rightarrow$ | 73.2 | 98.6 | $\rightarrow$ | 35.5 | 98.8 | $\rightarrow$ | 73.4 | 97.7 | $\rightarrow$ | 33.3
SVM [48] | 96.9 | $\rightarrow$ | 27.3 | 99.5 | $\rightarrow$ | 27.3 | 99.6 | $\rightarrow$ | 46.2 | 99.1 | $\rightarrow$ |  11.0 | 99.8 | $\rightarrow$ | 23.1 | 99.5 | $\rightarrow$ |  0.0 | 99.8 | $\rightarrow$ | 27.0 | 99.5 | $\rightarrow$ |  0.0
DNN [47] | 100 | $\rightarrow$ | 31.5 | 100 | $\rightarrow$ |  0.5 | 100 | $\rightarrow$ | 70.0 | 100 | $\rightarrow$ |  9.1 | 100 | $\rightarrow$ | 58.6 | 100 | $\rightarrow$ |  8.2 | 100 | $\rightarrow$ | 62.6 | 100 | $\rightarrow$ | 15.1
BU [52] | 89.9 | $\rightarrow$ | 33.5 | 60.7 | $\rightarrow$ |  0.0 | 88.5 | $\rightarrow$ | 18.0 | 73.2 | $\rightarrow$ |  0.0 | 61.9 | $\rightarrow$ | 35.9 | 9.1 | $\rightarrow$ |  0.0 | 93.0 | $\rightarrow$ | 32.8 | 73.1 | $\rightarrow$ |  5.0
Chest X-ray | MIM (Adv. Acc=98.1%) | TIM (Adv. Acc=85.8%) | PGD (Adv. Acc=90.9%) | CW (Adv. Acc=98.9%)
AUC | TPR@90 | AUC | TPR@90 | AUC | TPR@90 | AUC | TPR@90
KD [52] | 100 | $\rightarrow$ | 67.9 | 100 | $\rightarrow$ |  7.9 | 99.3 | $\rightarrow$ | 74.6 | 98.8 | $\rightarrow$ |  11.0 | 100 | $\rightarrow$ | 82.3 | 100 | $\rightarrow$ | 50.5 | 99.2 | $\rightarrow$ | 71.5 | 98.4 | $\rightarrow$ | 15.7
MAHA [18] | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ |  5.4 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ | 22.4 | 100 | $\rightarrow$ |  0.0
LID [17] | 100 | $\rightarrow$ | 47.5 | 100 | $\rightarrow$ |  2.3 | 99.6 | $\rightarrow$ | 61.1 | 99.7 | $\rightarrow$ | 28.6 | 100 | $\rightarrow$ | 49.1 | 100 | $\rightarrow$ |  1.5 | 99.2 | $\rightarrow$ | 64.5 | 98.4 | $\rightarrow$ | 14.4
SVM [48] | 100 | $\rightarrow$ | 8.9 | 100 | $\rightarrow$ | 46.7 | 98.5 | $\rightarrow$ | 6.9 | 99.6 | $\rightarrow$ |  2.3 | 100 | $\rightarrow$ |  5.8 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ | 21.2 | 100 | $\rightarrow$ |  0.0
DNN [47] | 100 | $\rightarrow$ | 35.5 | 100 | $\rightarrow$ |  1.0 | 100 | $\rightarrow$ | 55.6 | 100 | $\rightarrow$ |  0.9 | 100 | $\rightarrow$ | 33.7 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ | 61.6 | 100 | $\rightarrow$ |  5.2
BU [52] | 100 | $\rightarrow$ | 15.2 | 100 | $\rightarrow$ |  0.0 | 94.6 | $\rightarrow$ | 7.1 | 83.3 | $\rightarrow$ |  0.0 | 49.2 | $\rightarrow$ | 26.2 | 22.7 | $\rightarrow$ |  0.0 | 98.3 | $\rightarrow$ | 26.2 | 94.8 | $\rightarrow$ |  0.0
Brain | MIM (Adv. Acc=100%) | TIM (Adv. Acc=100%) | PGD (Adv. Acc=100%) | CW (Adv. Acc=100%)
AUC | TPR@90 | AUC | TPR@90 | AUC | TPR@90 | AUC | TPR@90
KD [52] | 100 | $\rightarrow$ | 50.8 | 100 | $\rightarrow$ | 14.5 | 100 | $\rightarrow$ | 56.3 | 100 | $\rightarrow$ | 13.5 | 100 | $\rightarrow$ | 61.5 | 100 | $\rightarrow$ | 23.1 | 100 | $\rightarrow$ | 60.6 | 100 | $\rightarrow$ | 14.3
MAHA [18] | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ |  0.0 | 100 | $\rightarrow$ | 0.0 | 100 | $\rightarrow$ |  0.0
LID [17] | 100 | $\rightarrow$ | 50.3 | 100 | $\rightarrow$ | 10.0 | 100 | $\rightarrow$ | 57.2 | 100 | $\rightarrow$ | 18.8 | 100 | $\rightarrow$ | 49.3 | 100 | $\rightarrow$ | 23.1 | 100 | $\rightarrow$ | 58.2 | 100 | $\rightarrow$ | 21.3
SVM [48] | 100 | $\rightarrow$ | 72.9 | 100 | $\rightarrow$ | 54.3 | 100 | $\rightarrow$ | 66.5 | 100 | $\rightarrow$ | 34.5 | 100 | $\rightarrow$ | 75.5 | 100 | $\rightarrow$ | 51.5 | 100 | $\rightarrow$ | 77.9 | 100 | $\rightarrow$ | 51.0
BU [52] | 100 | $\rightarrow$ | 45.8 | 100 | $\rightarrow$ | 8.6 | 100 | $\rightarrow$ | 35.4 | 100 | $\rightarrow$ |  7.6 | 100 | $\rightarrow$ | 43.7 | 100 | $\rightarrow$ | 1.5 | 100 | $\rightarrow$ | 38.5 | 100 | $\rightarrow$ | 2.7

##  4 Adversarial attack with a hierarchical feature constraint

In this section, we demonstrate a way of hiding the adversarial representation
in the target feature distribution. Our intuition is to derive a term that
measures the distance from the adversarial representation to this
distribution; therefore, the adversarial representation can be pushed toward
such a distribution on the shortest path by directly minimizing the defined
term during the process of stochastic gradient descent in each iteration of
adversarial attack.

Modeling the target feature distribution: First, the feature distribution of
the target class is modeled using a Gaussian mixture model (GMM) as follows:

$p(f_{\theta}^{l}(x))=\sum_{m=1}^{M}\pi_{m}\mathcal{N}(f_{\theta}^{l}(x)|\mu_{m},\Sigma_{m}),$
(14)

where $p$ is the probability density of sample $x$ with respect to the target
class $c$; $f_{\theta}^{l}(\cdot)$ denotes the mapping function, _e.g.,_ the
deep representation of the $l^{th}$ activation layer with parameters $\theta$;
$\pi_{m}$ is the mixture coefficient subject to $\sum_{m=1}^{M}\pi_{m}=1$; and
$\mu_{m}$ and $\Sigma_{m}$ are the mean and covariance matrix of the $m$-th
Gaussian component in the mixture model. We train these parameters by the
expectation-maximization (EM) algorithm [57] on the training images of the
target class $c$.

For a given input $x$, we separately compute the log-likelihood of an
adversarial feature relative to each component, and choose the most probable
Gaussian component:

$m^{\prime}=\mathop{\arg\max}\limits_{m}~{}\ln(\pi_{m}\mathcal{N}(f_{\theta}^{l}(x)|\mu_{m},\Sigma_{m})).$
(15)

Then, the log-likelihood of this chosen component $m^{\prime}$ is maximized to
hide the adversarial representation.

Algorithm 1 BIM attack with hierarchical feature constraint

0: Training set $(x^{i},y^{i})\in\mathcal{D}$, an input image $x$, target
class $c$, a DNN model $h$ with $L$ feature layers
$\\{f_{\theta}^{l};~{}l=1:L\\}$, the maximum number of iterations $T$, and a
step size $\alpha$

0: Adversarial image $x^{*}$ with $||x^{*}-x||_{\infty}\leq\epsilon$

1: for each DNN’s layer $l=1$ to $L$ do

2: Initialize mean $\mu_{m}^{l}$ and covariance $\Sigma_{m}^{l}$ of each
component, where $m\in\\{1,...,M\\}$

3: Train the GMM using the EM algorithm with $f_{\theta}^{l}(x^{i})$, where
$x^{i}\in\mathcal{D}$ with $y^{i}=c$

4: end for

5: $x_{0}^{*}=x$

6: for $t=1$ to $T$ do

7: $J_{final}=J_{\mathrm{HFC}}+\dot{J}$,

8:
$x_{t+1}^{*}=\mathrm{clip}_{\epsilon}(x_{t}^{*}-\alpha\cdot\mathrm{sign}(\nabla_{x}(J_{\mathrm{HFC}}+\dot{J})))$

9: end for

10: return $x^{*}$

Hierarchical feature constraint: To avoid being detected by outlier detectors,
we add the constraint
$\ln(\pi_{m}\mathcal{N}(f_{\theta}^{l}(x)|\mu_{m^{\prime}},\Sigma_{m^{\prime}}))$,
ignoring the constant terms, to all DNN layers. The hierarchical feature
constraint induces a loss ($J_{\mathrm{HFC}}$) that can be formulated as:

$J_{\mathrm{HFC}}=\sum^{L}_{l=1}\frac{\lambda^{l}}{2}(f_{\theta}^{l}(x)-\mu^{l}_{m^{\prime}(l)})^{\top}(\Sigma_{m^{\prime}(l)}^{l})^{-1}(f_{\theta}^{l}(x)-\mu^{l}_{m^{\prime}(l)}),$
(16)

where $\lambda^{l}$ is a weighting factor that controls the contribution of
constraint in layer $l$, $m^{\prime}(l)$ is the chosen mixture component for
layer $l$, and $L$ is the total number of layers.

Algorithm 1 shows the pseudo-code for the BIM attack with hierarchical feature
constraint. Given an input image $x$, the goal of our HFC is to find an
adversarial example that can be misclassified to the target class $c$, while
keeping the deep representation close to the feature distribution of normal
samples. Here, we focus on the AEs with the $L_{\infty}$ constraint.
Concretely, we first model the target hierarchical features of the training
data with GMM, and then extend the attacking process of BIM by replacing the
original loss function $J$ in Eq. (3) with:

$J_{\mathrm{final}}=J_{\mathrm{HFC}}+\dot{J},$ (17)

where $\dot{J}$ is the classification loss used in original attack (e.g., Eq.
(4) in CW attack and cross-entropy loss $J$ in PGD, BIM, MIM, etc) and
$J_{\mathrm{HFC}}$ is the HFC loss term. Furthermore, we can also easily apply
the HFC term to other advanced attacks (e.g., MIM, TIM, etc) by simply adding
HFC to the original loss function.

Figure 5: The AUC scores in the solid lines are the performances of the
adversarial detectors trained on AEs generated by BIM, where most of the
scores are around 100.0%, resulting in overlapped solid lines at the top of
the charts. The dotted lines denote the adversarial detection accuracy under
our proposed attack. The performance drop of each adversarial detector can be
observed from the pair of solid and dotted line of corresponding color. Our
method can bypass an array of adversarial detectors as indicated by a drastic
performance drop from solid line to dotted line on different backbones (left
for VGG-16 and right for ResNet-50), datasets (top for Fundoscopy and bottom
for Chest X-Ray), and perturbation budgets ($L_{\infty}/256$ is indicated by
the horizontal axis).

###  4.1 HFC for anomaly detection

Interestingly, we find that HFC has the potential to detect anomaly signals.
For the input $x$, we compute the HFC term of the feature in the last layer
$n$ as the image-level anomaly score $S_{\mathrm{HFC}}$ as follows:

$\displaystyle\vspace{-10mm}m^{\prime}$
$\displaystyle=\mathop{\arg\max}\limits_{m}~{}\ln(\pi_{m}\mathcal{N}(f_{\theta}^{n}(x)|\mu_{m},\Sigma_{m})).$
(18) $\displaystyle S_{\mathrm{HFC}}$
$\displaystyle=(f_{\theta}^{n}(x)-\mu^{i}_{m^{\prime}(i)})^{\top}(\Sigma_{m^{\prime}(n)}^{n})^{-1}(f_{\theta}^{n}(x)-\mu^{n}_{m^{\prime}(n)})$
(19)

where $f_{\theta}^{n}(x)$ is the feature of the last layer of input $x$;
$\mu_{m}$ and $\Sigma_{m}$ are the mean and covariance matrix of the $m$-th
component of GMM for the last layer; $m^{\prime}$ is the component of the GMM
model with the minimum log-likelihood.

##  5 Experiments

###  5.1 Setup

Evaluation process and metrics. The targeted adversarial example (AE) first
manipulates the results of the disease classifier $h$, and we report the
Adversarial accuracy (Adv. Acc) representing the success rate of AE in
manipulating $h$. Then, the reactive defense $d$ is supposed to identify the
AE that succeeds in attacking the classifier $h$. We report True positive rate
at 90% true negative rate (TPR@90): 10% of the normal samples are dropped by
the defense $d$ to reject more AEs; and Area under curve (AUC) score of
defense $d$ for detecting AEs. A strong attacker should manipulate classifier
$h$ (higher Adv. Acc) and compromise the reactive defense $d$ (lower TPR@90
and AUC) simultaneously.

Assumptions of the attackers. In the white-box setting, the attackers are
aware of the parameters of the disease classifier $h$, but not aware of any
specific strategy or parameters of the deployed reactive defense $d$. While in
the semi-white-box setting (in Section V.E), the attackers are only aware of
the type of the disease classifier $h$, but not aware of the parameters of $h$
as well as the specifics of the deployed reactive defense $d$. For the
computational capabilities, we assume that the attacker has a GPU to generate
AEs using HFC.

Datasets. We choose two public datasets on typical medical classification
tasks with 2D images, following [11, 10]. (i) Kaggle Fundoscopy [53] dataset
on the diabetic retinopathy (DR) diagnosis task, consisting of 3,663 high-
resolution fundus images. Each image is labeled to one of the five levels from
‘No DR’ to ‘mid/moderate/severe/proliferate DR’. Following [11, 10], we
conduct a binary classification experiment, which considers all fundus images
labeled with DR as the same class. (ii) Kaggle555Kaggle Chest X-ray Images
(Pneumonia): https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia
dataset on the pneumonia diagnosis task, which contains 5,863 Chest X-ray
images labeled with ‘Normal’ or ‘Pneumonia’ .

Furthermore, we evaluate HFC on a 3D medical volume dataset: Brain hemorrhage
(Brain), which contains 1,486 brain CT volumes collected from our
collaborative hospital with Institutional Review Boards (IRB) approval. The
volumes are labeled according to the pathological causes of cerebral
hemorrhage: aneurysm, arteriovenous malformation, moyamoya disease, and
hypertension. Each CT volume is standardized to 30$\times$270$\times$270
voxels. We randomly crop patches with a size of 30$\times$128$\times$128 and
normalize them to [0,1] during training.

Following [11, 17], each dataset is split into three subsets: Train, AdvTrain
and AdvTest. Also, we randomly select 80% of the images as Train set to train
the DNN classifier, and the left samples are treated as the Test set. We
discard the misclassified test samples (by the diagnosis network), dropping 7,
101, 50 samples on Fundoscopy, Chest X-ray, and Brain datasets, respectively.
Then the 70% of the samples (AdvTrain) in the Test set are used to train the
adversarial detectors, and the left 30% images (AdvTest) are used to evaluate
their effectiveness.

DNN classifiers. For 2D medical images (Fundoscopy and Chest X-ray), the
ResNet-50 [16] and VGG-16 [58] models pretrained on ImageNet [56] are chosen
as classifier. All images are resized to 256$\times$256$\times$3 and
normalized to [0,1]. The models are trained with augmented data using random
crop and horizontal flip. For 3D medical volumes, we choose ResNet-18 [16] as
our backbone by replacing the 2D convolution kernels with 3D kernels.

Adversarial attacks and detectors. Following [11], we choose MIM, BIM, PGD,
TIM [28] and CW to attack our models. For the adversarial detectors, we choose
the following reactive defenses:

  * •

Kernel density (KD) [52]. Given a sample $x$ of class $k$, and a set of
training samples from the same class $X_{k}$, the KD score of $x$ can be
estimated by:

$\mathrm{KD}(x)=\sum_{x^{\prime}\in X_{k}}F(z(x),z(x^{\prime})),$ (20)

where $z(\cdot)$ is the activation vector of the penultimate layer,
$F(\cdot,\cdot)$ is the kernel function, often chosen as a Gaussian kernel.

  * •

Bayesian uncertainty (BU) [52]. Given a sample $x$, the BU score of $x$ can be
estimated by:

$\mathrm{BU}(x)=\frac{1}{N}\sum_{n=1}^{T}{l_{n}}^{T}l_{n}-(\frac{1}{N}\sum_{n=1}^{T}{l_{n}})^{T}(\frac{1}{T}\sum_{n=1}^{T}{l_{n})},$
(21)

where $\\{{l_{1}},...,{l_{N}}\\}$ are the stochastic logits predictions
calculated when randomly dropping out 30% features of the penultimate layer.
Here, we set $N=50$.

  * •

Local intrinsic dimensionality (LID) [17] describes the rate of expansion in
the number of data objects as the distance from the reference sample
increases. Specifically, given a sample $x$, LID makes use of its distances to
the first $N$ nearest neighbors:

$\mathrm{LID}(f_{l}(x))=-(\frac{1}{N}\sum_{i=n}^{N}log\frac{r_{n}(f_{l}(x))}{r_{N}(f_{l}(x))})^{-1},$
(22)

where $f_{l}(x)$ is the activation values from the $l^{th}$ intermediate
layer; $r_{n}(f_{l}(x))$ is the Euclidean distance between $f_{l}(x)$ and its
$n^{th}$ nearest neighbor. LID is computed on each activation layer of the
diagnosis network.

  * •

Mahalanobis distance-based confidence score (MAHA) [18] utilizes the
Mahalanobis distance-based metric to measure the outlier degree of the
features. Specifically, we first compute the empirical mean $\mu_{l}$ and
covariance $\Sigma_{l}$ of the activations for each layer $l$ of the training
samples. Then, we compute the MAHA as:

$\mathrm{MAHA}(f_{l}(x))=(f_{l}(x)-\mu_{l})^{\top}\Sigma_{l}^{-1}(f_{l}(x)-\mu_{l}),$
(23)

where $f_{l}(\cdot)$ is the feature in the $l^{th}$ activation layer of DNN.
MAHA is computed on each layer of the diagnosis network.

  * •

RBF-SVM (SVM) [48] is a learning-based adversarial detector using RBF-SVM
trained on the normal and adversarial features from the penultimate layer.

  * •

Deep neural network (DNN) [47] is a learning-based adversarial detector using
DNN trained on the normal and adversarial features. We train DNN on each
activation layer of the diagnosis network and ensemble these networks by
summing up their logits.

Table 2: Comparison of other adaptive attacks aiming at manipulating
representations. The AEs are generated on ResNet-50 under the constraint
$L_{\infty}=1/256$ using BIM. We choose AUC scores (%) and ”Adv. Acc” (%,
$\uparrow$) as metrics.

Fundoscopy | KD | MAHA | LID | SVM | DNN | Adv. Acc
---|---|---|---|---|---|---
Random [20] | 75.1 | 86.1 | 91.7 | 48.2 | 93.7 | 100.0
Closest [20] | 77.0 | 64.0 | 91.0 | 13.0 | 79.3 | 81.5
KDE [22] | 51.6 | 86.5 | 90.9 | 45.3 | 95.0 | 100.0
LID [21] | 87.6 | 85.4 | 93.4 | 61.2 | 96.2 | 95.9
HFC | 74.2 | 6.4 | 78.3 | 28.6 | 60.0 | 99.5
Chest X-ray | KD | MAHA | LID | SVM | DNN | Adv. Acc
Random [20] | 77.0 | 64.0 | 91.0 | 13.0 | 79.3 | 94.8
Closest [20] | 80.1 | 38.3 | 71.3 | 9.0 | 87.7 | 53.1
KDE [22] | 58.2 | 66.9 | 71.7 | 15.3 | 95.6 | 100.0
LID [21] | 84.0 | 66.6 | 77.1 | 28.6 | 96.6 | 70.9
HFC | 70.8 | 0.0 | 53.6 | 16.7 | 32.6 | 95.8

The parameters for KD, LID, BU, and MAHA are set according to the original
papers. We compute the scores of LID and MAHA for all of the activation layers
and train a logistic regression classifier [18, 17]. We implement KD, LID, BU,
and MAHA using the official code.666 KD and BU,
https://github.com/rfeinman/detecting-adversarial-samples
LID and MAHA, https://github.com/pokaxpoka/deep_Mahalanobis_detector

Hyperparameters. We set $M=16$ and $64$ for Chest X-ray and Fundoscopy
datasets, respectively. We compute the mean value of the features of each
channel in the $l^{th}$ activation layer separately. We set $\lambda_{l}$ to
$\frac{1}{C_{l}}$, where $C_{l}$ is the number of channels. As a small
adversarial perturbation in medical images increases the loss drastically
[11], we set a tiny step size $\alpha=0.02/256$ and the number of iteration
$T$ to $2\epsilon/\alpha$.

###  5.2 Bypassing adversarial detectors

Figure 6: The AUC scores in the solid lines are the performances of the
adversarial detectors for each activation layer (from ResNet-50 on
Fundoscopy). Our HFC compromises the detection accuracy (the gaps between
solid lines and the corresponding dotted lines) at each layer.

We compare the performances of the existing adversarial attacks with and
without HFC on different datasets, perturbation constraints, and DNN
classifiers. The 2D t-SNE in Fig. 7 illustrates that HFC successfully moves
the adversarial features from outlier (orange) regions to the normal regions
(cyan) where target features (purple) reside. As shown in Fig. 6, the SOTA
detectors from all of the activation layers in the diagnosis network are
compromised by HFC, _e.g.,_ the AUC scores of KD and LID are compromised from
nearly 100% to less than 80%. Moreover, HFC can decrease the MAHA scores to
less than clean samples (as the AUC scores decreased to $<$ 20%) by pushing
adversarial features closer to the empirical mean $\mu_{l}$ in Eq. (23).

Quantitative results in Table 1 justify that four widely-used adversarial
attacks are boosted by HFC to bypass an array of SOTA detectors simultaneously
with a high accuracy, under the small perturbation constraint
$L_{\infty}=1/256$. On the three datasets covering both 2D and 3D modalities,
binary and multi-class settings, most AUC scores of the detectors are
decreased to below 80%. Furthermore, when we increase the perturbation budget,
the HFC is provided more room to manipulate the representations, compromising
the detectors more drastically (as the dotted lines shown in Fig. 5 decrease).
Please note that, as different detectors computes anomaly scores in different
ways, HFC may not make sure the anomaly scores of AEs are lower than clean
samples. For the detectors with AUC scores $<$ 0.5, the decreasing trend may
be irregular.

###  5.3 Comparison of other adaptive attacks

We compare the effectiveness of our HFC with other competitive adaptive
attacks, designed to manipulate the features and bypass the detectors: 1)
Generate AEs with internal features similar to a randomly selected guide image
[20]; 2) Instead of random sampling, select a guide image with its feature
closest to the input [20]; 3) Minimize cross-entropy and the loss term of KDE
(Eq. (20)) at the same time [22]; 4) Minimize cross-entropy and the loss term
of LID (Eq. (22)) at the same time [21]. The results in Table 2 show that, all
attacks trying to manipulate the adversarial features can successfully weaken
the ability of the SOTA detectors. Meanwhile, under the strict constraint
($L_{\infty}=1/256$), our proposed HFC greatly outperforms other strong
adaptive attacks, especially on compromising MAHA, LID and DNN.

Figure 7: The 2D t-SNE visualization of the features of clean sample and
adversarial samples generated from BIM and our method HFC. Features are
extracted from ResNet-50 on Chest X-ray. The adversarial examples generated by
our method are surrounded by the clean data in the feature space. Table 3: The
AUC (%) on Fundoscopy dataset of different adversarial detectors on
adversarial examples generated by the proposed method with different numbers
(M) of GMM components to model the clean data distribution. Smaller numbers
indicate better performance.

$M$ | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128
---|---|---|---|---|---|---|---|---
KD | 75.8 | 78.5 | 77.9 | 77.8 | 73.6 | 73.3 | 74.2 | 74.4
MAHA | 5.9 | 5.6 | 6.2 | 7.1 | 8.2 | 8.6 | 7.6 | 5.9
LID | 81.7 | 82.4 | 83.0 | 84.2 | 83.2 | 80.7 | 78.3 | 78.7
SVM | 52.3 | 45.7 | 40.4 | 34.0 | 31.4 | 34.5 | 35.5 | 40.3
DNN | 61.4 | 60.7 | 62.6 | 64.3 | 64.4 | 63.5 | 63.4 | 64.9
BU | 55.5 | 52.2 | 49.7 | 45.2 | 43.3 | 42.6 | 43.8 | 42.6
Average | 55.4 | 54.2 | 53.3 | 57.8 | 50.7 | 50.5 | 50.4 | 51.3
Maximum | 81.7 | 82.4 | 83.0 | 84.2 | 83.2 | 80.7 | 78.3 | 78.7

###  5.4 Hyperparameter analysis

We model the ResNet-50 feature distributions of normal samples on the
Fundoscopy dataset by GMM with different numbers of components and evaluate
the corresponding attacking performances. As shown in Table 3, all of the
attacks can stably compromise the detectors, while setting $M=64$ can slightly
improve the performance. We speculate that accommodating an excessive number
of components within the constrained size of clean samples could pose
challenges to the proper fitting of GMMs. Conversely, an insufficient number
of components produces a rough fit of the underlying distribution, which might
impede the shortest path convergence to push the adversarial feature to the
clean side.

Figure 8: The visualization of synthesized OOD images. We mark the OOD signals with red boxes, which are “lesions” generated by random shapes and textures [59, 60]. “Tiny, Small, Medium, and Large” are the sizes of lesions, while the numbers are the quantities of the lesions. Table 4: The performance of BIM under the semi-white-box setting w/ and w/o HFC. The AEs are crafted under constraint $L_{\infty}=4/256$. AUC scores (%, $\downarrow$) and ”Adv. Acc” (%, $\uparrow$) are used as metric. DNN Model | Attacks | Fundoscopy [53] | Chest X-ray [55]
---|---|---|---
KD | MAHA | LID | SVM | DNN | Adv. Acc | KD | MAHA | LID | SVM | DNN | Adv. Acc
ResNet-50 | BIM | 98.7 | 100.0 | 99.5 | 92.1 | 100.0 | 83.2 | 92.6 | 95.4 | 79.5 | 24.1 | 99.5 | 85.8
BIM w/ HFC | 78.0 | 9.1 | 68.0 | 16.9 | 43.8 | 68.2 | 89.4 | 85.9 | 71.5 | 11.4 | 79.5 | 76.4
VGG-16 | BIM | 72.3 | 89.6 | 81.5 | 48.1 | 95.2 | 88.6 | 46.4 | 89.2 | 81.3 | 73.6 | 98.5 | 98.3
BIM w/ HFC | 50.9 | 18.2 | 64.6 | 28.8 | 16.7 | 73.2 | 34.6 | 8.6 | 49.4 | 41.3 | 69.5 | 88.9

###  5.5 Semi-white-box attack

We evaluate the proposed HFC in a more difficult scenario: only the
architecture information of DNN models is available. The attacker aims at
confusing the victim model and bypassing its adversarial detectors
simultaneously, without accessing the parameters of the victim model. In the
experiments, we train substitute diagnosis networks and detectors using the
same training data and generate AEs using BIM [19] w/ and w/o HFC. As shown in
Table 4, our HFC is able to help BIM bypass most of the detectors and control
the prediction of the victim model at the same time, posing more disturbing
concerns to the safety of DNNs for further clinical application.

Table 5: The comparison between the attacks in the medical and natural image domains using HFC. The adversarial attacks are deployed on ResNet-50. AUC scores (%, $\downarrow$) are used as metrics for adversarial detectors. CIFAR-10 | KD | MAHA | LID | SVM | DNN
---|---|---|---|---|---
BIM$|L_{\infty}$=8/256 | 76.4 | 91.4 | 80.8 | 96.9 | 99.8
Ours$|L_{\infty}$=8/256 | 30.1 | 82.7 | 58.7 | 87.9 | 87.3
CIFAR-10 | KD | MAHA | LID | SVM | DNN
BIM$|L_{\infty}$=16/256 | 87.5 | 99.0 | 90.3 | 99.0 | 100.0
Ours$|L_{\infty}$=16/256 | 24.0 | 76.4 | 54.1 | 85.8 | 80.7
Chest X-ray | KD | MAHA | LID | SVM | DNN
BIM$|L_{\infty}$=1/256 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0
Ours$|L_{\infty}$=1/256 | 71.3 | 0.0 | 48.6 | 16.7 | 31.8

###  5.6 Comparing Chest X-ray and CIFAR-10 attacks

We also evaluate the Chest X-ray performance of the adversarial detectors and
HFC under constraints $L_{\infty}=8/256$ and $16/256$ on CIFAR-10 [54], a
natural image dataset. The Adv Acc. will be less than 95% for Chest X-ray and
CIFAR-10 when the perturbation is smaller than $0.5/256,8/256$, respectively.
As shown in Table 5, a greater perturbation budget changes the adversarial
features more drastically in the similar direction to the outlier, which makes
the detectors easier to detect. At the same time, a larger perturbation budget
gives HFC more room to manipulate the deep representation and enhance its
ability to compromise the adversarial detectors more drastically.

On the other hand, as illustrated in Sec. 3.2, the Chest X-ray image
classification models are more vulnerable than CIFAR-10 ones. As a
consequence, HFC enjoys more success in manipulating the adversarial feature
into the feature distribution of normal samples even with a small
perturbation. Compared to CIFAR-10 natural images, when the attacker utilizes
HFC to camouflage Chest X-ray medical adversarial attacks, greater performance
drops of the adversarial detectors can be found in Table 5. This discovery
suggests that improving the robustness of medical models is necessary.

Figure 9: The comparison of the successful rate (Adv. Acc, $\uparrow$) of 3D
and 2D medical images. We choose BIM to attack ResNet-18-3D, ResNet-50 for 3D,
2D images, respectively. Greater Adv. means more vulnerability.

###  5.7 Comparison 3D and 2D vulnerability

In addition, we compare the vulnerability of 2D and 3D medical volumes.
Specifically, we attack the disease classification network with BIM within
tiny perturbation budgets ($L_{\inf}=[0.01\sim 0.2]/256$). As shown in Fig. 9,
BIM has a greater successful rate on 3D volumes than 2D ones, especially
within $L_{\inf}=0.02/256$, which means the 3D medical volumes are likely more
vulnerable than 2D images.

Analysis: According to Goodfellow’s linear explanation [27], sufficient
dimensionality is the primary source of CNN vulnerability to adversarial
attacks. We speculate that increasing the dimensionality of input images
likely makes neural networks more susceptible to adversarial perturbations,
and the larger dimensionality of 3D images brings greater vulnerability. In
our experiments, the input dimensionality for 3D images is $30\times 270\times
270$, while for 2D images, it is $256\times 256\times 3$, which is
significantly smaller. We show that 3D input is more vulnerable. To better
validate this hypothesis, we design an experiment to evaluate the robustness
by reducing the dimensionality of the 3D input. As illustrated in Fig. 9, when
we 2x-downsample the 3D images and reduce the overall 3D input dimensionality
by $8\times$, a significant improvement in adversarial robustness is observed,
which somewhat confirms the linear explanation [27] hypothesis.

Table 6: The AUC scores (%, $\uparrow$) represent the performances of HFC term to detect the OOD signals, which are various sizes and quantities of synthesized “lesions”. FR (%) denotes the false rates of the classification network influenced by OOD signals, leading to incorrect predictions. Lesions | Fundoscopy | Chest X-ray
---|---|---
Size | Quantity | AUC(%) | FR(%) | AUC(%) | FR(%)
small | 2 | 87.4 | 0.2 | 94.4 | 1.2
small | 4 | 92.4 | 0.6 | 97.7 | 4.3
small | 6 | 94.6 | 0.7 | 98.1 | 10.0
medium | 1 | 84.9 | 0.2 | 94.8 | 0.5
medium | 2 | 91.4 | 0.2 | 97.9 | 1.2
medium | 4 | 95.5 | 0.5 | 99.5 | 3.4
large | 1 | 91.9 | 0.2 | 98.1 | 2.1
large | 2 | 94.9 | 0.4 | 99.6 | 4.3
large | 3 | 97.2 | 0.5 | 99.7 | 7.5

###  5.8 Detecting OOD signals using HFC term

In this section, we try to evaluate the possibility of using HFC term to
detect the out-of-distribution (OOD) outlier data. Because obtaining the real-
world OOD signals (e.g., artifacts, anatomical variations, or unseen
pathologies) is difficult, we modify lesion synthesis methods proposed by Hu
et al. [59, 60] to simulate OOD images by synthesizing different sizes and
quantities of “lesions” and insert into a random position in clean images,
which are served as anomaly signals and shown in Fig. 8. Specifically, the
“lesion” generation process employs random elastic deformations to deform
randomly-sized ellipses and generate various shapes. Additionally, Gaussian
filtering with random parameters is applied to filter salt noise to generate
various textures, which are then incorporated into the ellipses. Finally,
blurring is applied to the edges.

Next, we respectively compute the anomaly scores $S_{\mathrm{HFC}}$ (in Eq.
19) for clean images and OOD images on the test set, as well as AUC scores
accordingly. Given a certain lesion quantity and type, we randomly synthesize
anomalies ten times for each image. For simplicity, we ignore the wrong
predicted OOD images caused by the synthesized mask, the false-prediction
rates (FR) caused by the OOD signals are shown in Table 6. Moreover, we change
the sizes and quantities of lesions to control the degree of outlier.
Surprisingly, as the results in Table 6, we find that our HFC does have the
potential to detect the possible outliers with AUC scores larger than 85%,
even the small OOD signal such as small lesions. Furthermore, HFC exhibits a
stronger detection capability for more potent OOD signals (larger and more
numerous lesions). This finding inspires a strong OOD detection method for
more real-world OOD signals in the future.

###  5.9 Limitation

It is a limitation of our method to directly apply on auto-PGD (APGD) [61],
which auto-decays the step size (also named as learning rate) of each
iteration in PGD according to the gradient. Therefore, directly adding the HFC
term on the original cross-entropy loss disrupts the original automatic decay
mechanism of APGD.

##  6 Conclusion

In this paper, we investigate the characteristics of medical adversarial
examples in feature space. It is theoretically proved that existing
adversarial attacks tend to optimize the features in a similar direction. In
consequence, the adversarial features become outliers and easy to detect. A
stress test is then performed to reveal the greater vulnerability of medical
image features, compared to the natural ones. However, this vulnerability can
be exploited by the attacker to hide the adversarial features. We herein
propose a novel hierarchical feature constraint (HFC), a simple-yet-effective
add-on that can be applied to any existing attack, to avoid adversarial
examples from being detected. The effectiveness of HFC is validated by
extensive experiments, which also significantly surpasses other adaptive
attacks. HFC reveals the limitation of the current defenses for detecting
medical adversarial attacks in the feature space and allows to develop the
importance of the robustness of the medical features. We hope it can inspire
stronger defenses in the future.

## References

  * [1] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in _International Conference on Learning Representations_ , 2014.
  * [2] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boosting adversarial attacks with momentum,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9185–9193.
  * [3] S. K. Zhou, H. Greenspan, C. Davatzikos, J. S. Duncan, B. van Ginneken, A. Madabhushi, J. L. Prince, D. Rueckert, and R. M. Summers, “A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises,” _Proceedings of the IEEE_ , 2021.
  * [4] N. Mangaokar, J. Pu, P. Bhattacharya, C. K. Reddy, and B. Viswanath, “Jekyll: Attacking medical image diagnostics using deep generative models,” in _2020 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE, 2020, pp. 139–157.
  * [5] S. K. Zhou, Ed., _Medical image recognition, segmentation and parsing: machine learning and multiple object approaches_. Academic Press, 2015.
  * [6] S. K. Zhou, H. Greenspan, and D. Shen, Eds., _Deep learning for medical image analysis_. Academic Press, 2017.
  * [7] S. K. Zhou, D. Rueckert, and G. Fichtinger, _Handbook of medical image computing and computer assisted intervention_. Academic Press, 2019.
  * [8] U. Ozbulak, A. Van Messem, and W. De Neve, “Impact of adversarial examples on deep learning models for biomedical image segmentation,” in _Medical Image Computing and Computer Assisted Intervention_. Springer, 2019, pp. 300–308.
  * [9] M. Paschali, S. Conjeti, F. Navarro, and N. Navab, “Generalizability vs. robustness: investigating medical imaging networks using adversarial examples,” in _Medical Image Computing and Computer Assisted Intervention_. Springer, 2018, pp. 493–501.
  * [10] S. G. Finlayson, H. W. Chung, I. S. Kohane, and A. L. Beam, “Adversarial attacks against medical deep learning systems,” _Science_ , vol. 363(6433), pp. 1287–1289, 2018.
  * [11] X. Ma, Y. Niu, L. Gu, Y. Wang, Y. Zhao, J. Bailey, and F. Lu, “Understanding adversarial attacks on deep learning based medical image analysis systems,” _Pattern Recognition_ , vol. 110, p. 107332, 2021.
  * [12] Q. Yao, Z. He, H. Han, and S. K. Zhou, “Miss the point: Targeted adversarial attack on multiple landmark detection,” in _Medical Image Computing and Computer Assisted Intervention_. Springer, 2020, pp. 692–702.
  * [13] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _International Conference on Learning Representations_ , 2018.
  * [14] X. Li and D. Zhu, “Robust detection of adversarial attacks on medical images,” in _IEEE International Symposium on Biomedical Imaging_. IEEE, 2020, pp. 1154–1158.
  * [15] L. v. d. Maaten and G. Hinton, “Visualizing data using t-SNE,” _Journal of Machine Learning Research_ , vol. 9, no. Nov, pp. 2579–2605, 2008\.
  * [16] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [17] X. Ma, B. Li, Y. Wang, S. M. Erfani, S. Wijewickrema, G. Schoenebeck, D. Song, M. E. Houle, and J. Bailey, “Characterizing adversarial subspaces using local intrinsic dimensionality,” in _International Conference on Learning Representations_ , 2018.
  * [18] K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified framework for detecting out-of-distribution samples and adversarial attacks,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 7167–7177.
  * [19] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial machine learning at scale,” in _International Conference on Learning Representations_ , 2017\.
  * [20] S. Sabour, Y. Cao, F. Faghri, and D. J. Fleet, “Adversarial manipulation of deep representations,” in _IEEE Symposium on Security and Privacy_ , 2016\.
  * [21] A. Athalye, N. Carlini, and D. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” in _International Conference on Learning Representations_ , 2018.
  * [22] N. Carlini and D. Wagner, “Adversarial examples are not easily detected: Bypassing ten detection methods,” in _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , 2017, pp. 3–14.
  * [23] Q. Yao, Z. He, Y. Lin, K. Ma, Y. Zheng, and S. K. Zhou, “A hierarchical feature constraint to camouflage medical adversarial attacks,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2021, pp. 36–47.
  * [24] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “DeepFool: a simple and accurate method to fool deep neural networks,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2574–2582.
  * [25] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _IEEE Symposium on Security and Privacy_ , 2017, pp. 39–57.
  * [26] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _International Conference on Learning Representations_ , 2015.
  * [27] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in _International Conference on Learning Representations_ , 2015.
  * [28] Y. Dong, T. Pang, H. Su, and J. Zhu, “Evading defenses to transferable adversarial examples by translation-invariant attacks,” in _CVPR_ , 2019, pp. 4312–4321.
  * [29] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, “Distillation as a defense to adversarial perturbations against deep neural networks,” in _IEEE Symposium on Security and Privacy_ , 2016, pp. 582–597.
  * [30] W. Xu, D. Evans, and Y. Qi, “Feature squeezing: Detecting adversarial examples in deep neural networks,” in _Network and Distributed System Security Symposium_ , 2017.
  * [31] F. Liao, M. Liang, Y. Dong, T. Pang, X. Hu, and J. Zhu, “Defense against adversarial attacks using high-level representation guided denoiser,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1778–1787.
  * [32] G. K. Dziugaite, Z. Ghahramani, and D. M. Roy, “A study of the effect of JPG compression on adversarial images,” _arXiv preprint arXiv:1608.00853_ , 2016\.
  * [33] A. S. Ross and F. Doshi-Velez, “Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2017.
  * [34] M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and N. Usunier, “Parseval networks: Improving robustness to adversarial examples,” in _ICML_. PMLR, 2017, pp. 854–863.
  * [35] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in _ACM Computer and Communications Security_ , 2017, pp. 506–519.
  * [36] X. Liu, M. Cheng, H. Zhang, and C.-J. Hsieh, “Towards robust neural networks via random self-ensemble,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 369–385.
  * [37] G. S. Dhillon, K. Azizzadenesheli, Z. C. Lipton, J. Bernstein, J. Kossaifi, A. Khanna, and A. Anandkumar, “Stochastic activation pruning for robust adversarial defense,” in _International Conference on Learning Representations_ , 2018.
  * [38] S. A. Taghanaki, K. Abhishek, S. Azizi, and G. Hamarneh, “A kernelized manifold mapping to diminish the effect of adversarial perturbations,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 340–11 349.
  * [39] X. He, S. Yang, G. Li, H. Li, H. Chang, and Y. Yu, “Non-local context encoder: Robust biomedical image segmentation against adversarial attacks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 8417–8424.
  * [40] Y. Dong, Q.-A. Fu, X. Yang, T. Pang, H. Su, Z. Xiao, and J. Zhu, “Benchmarking adversarial robustness on image classification,” in _CVPR_ , 2020, pp. 321–331.
  * [41] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, “Ensemble adversarial training: Attacks and defenses,” in _International Conference on Learning Representations_ , 2018.
  * [42] F. Tramer, N. Carlini, W. Brendel, and A. Madry, “On adaptive attacks to adversarial example defenses,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 1633–1645, 2020.
  * [43] D. Meng and H. Chen, “MagNet: a two-pronged defense against adversarial examples,” in _Proceedings of the ACM SIGSAC Conference on Computer and Communications Security_ , 2017, pp. 135–147.
  * [44] D. J. Miller, Z. Xiang, and G. Kesidis, “Adversarial learning targeting deep neural network classification: A comprehensive review of defenses against attacks,” _Proceedings of the IEEE_ , vol. 108, no. 3, pp. 402–433, 2020\.
  * [45] Z. Zheng and P. Hong, “Robust detection of adversarial attacks by modeling the intrinsic properties of deep neural networks,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 7913–7922.
  * [46] X. Li and F. Li, “Adversarial examples detection in deep networks with convolutional filter statistics,” in _ICCV_ , 2017, pp. 5764–5772.
  * [47] J. H. Metzen, T. Genewein, V. Fischer, and B. Bischoff, “On detecting adversarial perturbations,” in _International Conference on Learning Representations_ , 2017.
  * [48] J. Lu, T. Issaranon, and D. Forsyth, “Safetynet: Detecting and rejecting adversarial examples robustly,” in _ICCV_ , 2017, pp. 446–454.
  * [49] A. Dubey, L. v. d. Maaten, Z. Yalniz, Y. Li, and D. Mahajan, “Defense against adversarial images using web-scale nearest-neighbor search,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 8767–8776.
  * [50] N. Papernot and P. McDaniel, “Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning,” _arXiv preprint arXiv:1803.04765_ , 2018.
  * [51] G. Cohen, G. Sapiro, and R. Giryes, “Detecting adversarial samples using influence functions and nearest neighbors,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14 453–14 462.
  * [52] R. Feinman, R. R. Curtin, S. Shintre, and A. B. Gardner, “Detecting adversarial samples from artifacts,” _arXiv preprint arXiv:1703.00410_ , 2017\.
  * [53] Kaggle, “APTOS 2019 Blindness Detection,” 2019, https://www.kaggle.com/c/aptos2019-blindness-detection.
  * [54] A. Krizhevsky, “Learning multiple layers of features from tiny images,” _University of Toronto_ , 05 2012.
  * [55] D. S. Kermany, M. Goldbaum, W. Cai, C. C. Valentim, H. Liang, S. L. Baxter, A. McKeown, G. Yang, X. Wu, F. Yan _et al._ , “Identifying medical diagnoses and treatable diseases by image-based deep learning,” _Cell_ , vol. 172, no. 5, pp. 1122–1131, 2018.
  * [56] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2009.
  * [57] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” _Journal of the Royal Statistical Society: Series B (Methodological)_ , vol. 39, no. 1, pp. 1–22, 1977\.
  * [58] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations_ , 2015.
  * [59] Q. Hu, Y. Chen, J. Xiao, S. Sun, J. Chen, A. L. Yuille, and Z. Zhou, “Label-free liver tumor segmentation,” in _CVPR_ , 2023, pp. 7422–7432.
  * [60] Q. Yao, L. Xiao, P. Liu, and S. K. Zhou, “Label-free segmentation of covid-19 lesions in lung ct,” _IEEE transactions on medical imaging_ , vol. 40, no. 10, pp. 2808–2819, 2021.
  * [61] F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” in _ICML_. PMLR, 2020, pp. 2206–2216.