Adversarial Sample Detection Through Neural Network Transport Dynamics
We propose a detector of adversarial samples that is based on the view of
neural networks as discrete dynamic systems. The detector tells clean inputs
from abnormal ones by comparing the discrete vector fields they follow through
the layers. We also show that regularizing this vector field during training
makes the network more regular on the data distribution's support, thus making
the activations of clean inputs more distinguishable from those of abnormal
ones. Experimentally, we compare our detector favorably to other detectors on
seen and unseen attacks, and show that the regularization of the network's
dynamics improves the performance of adversarial detectors that use the
internal embeddings as inputs, while also improving test accuracy.
Skander Karkar, Patrick Gallinari, Alain Rakotomamonjy
# Adversarial Sample Detection Through Neural Network Transport Dynamics

###### Abstract

We propose a detector of adversarial samples that is based on the view of neural networks as discrete dynamic systems. The detector tells clean inputs from abnormal ones by comparing the discrete vector fields they follow through the layers. We also show that regularizing this vector field during training makes the network more regular on the data distribution's support, thus making the activations of clean inputs more distinguishable from those of abnormal ones. Experimentally, we compare our detector favorably to other detectors on seen and unseen attacks, and show that the regularization of the network's dynamics improves the performance of adversarial detectors that use the internal embeddings as inputs, while also improving test accuracy.

Keywords:Deep learning Adversarial detection Optimal transport

## 1 Introduction

Neural networks have improved performances on many tasks, including image classification. They are however vulnerable to adversarial attacks which modify an image in a way that is imperceptible to a human but that fools the network into wrongly classifying the image [50]. These adversarial images transfer between networks [39], can be carried out physically (e.g. causing autonomous cars to misclassify road signs [15]), and can be generated without access to the network [34]. Developing networks that are robust to adversarial samples or accompanied by detectors that can detect them is indispensable to deploying them safely [3].

We focus on detecting adversarial samples. Networks trained with a softmax classifier produce overconfident predictions even for out-of-distribution inputs [42]. This makes it difficult to detect such inputs via the softmax outputs. A detector is a system capable of predicting if an input at test time has been adversarially modified. Detectors are trained on a dataset made up of clean and adversarial inputs, after the network training. While simply training the detector on the inputs has been tried, using their intermediate embeddings works better [9]. Detectors vary by which activations to use and how to process them to extract the features that the classifier uses to tell clean samples from adversarial ones.

We make two contributions. First, we propose an adversarial detector that is based on the view of neural networks as dynamical systems that move inputs
in space, time represented by depth, to separate them before applying a linear classifier [55]. Our detector follows the trajectory of samples in space, through time, to differentiate clean and adversarial images. The statistics that we extract are the positions of the internal embeddings in space approximated by their norms and cosines to a fixed vector. Given their resemblance to the Euler scheme for differential equations, residual networks [21, 22, 55] are particularly amenable to this analysis. Skip connections and residuals are basic building blocks in many architectures such as EfficientNet [51] and MobileNetV2 [47], and ResNets and their variants such as WideResNet [60] and ResNeXt [59] remain competitive [57]. Visions Transformers [35, 14] are also mainly made up of residual stages. Besides, [58] show an increased vulnerability of residual-type architectures to transferable attacks, precisely because of the skip connections. This motivates the need for a detector that is well adapted to residual-type architectures. But the analysis and implementation can extend immediately to any network where most layers have the same input and output dimensions.

Our second contribution is to use the transport regularization during training proposed in [26] to make the activations of adversarial samples more distinguishable from those of clean samples, thus making adversarial detectors perform better, while also improving generalization. We prove that the regularization achieves this by making the network more regular on the support of the data distribution. This does not necessarily make it more robust, but it will make the activations of the clean samples closer to each other and further from those of out-of-distribution samples, thus making adversarial detection easier. This is illustrated on a 2-dimension example in Figure 1.

## 2 Related Work

Given a classifier \(f\) in a classification task and \(\epsilon\)\(>\)0, an adversarial sample \(y\) constructed from a clean sample \(x\) is \(y=x+\delta\), such that \(f(y)\neq f(x)\) and \(\|\delta\|_{p}\leq\epsilon\) for a certain \(L_{p}\) norm. The maximal perturbation size \(\epsilon\) has to be so small as to be almost imperceptible to a human. Adversarial attacks are algorithms that find such adversarial samples, and they have been particularly successful against neural networks [50, 8]. We present the adversarial attacks we use in our experiments in Appendix 0.D.1. The main defense mechanisms are robustness, i.e. training a network that is not easily fooled by adversarial samples, and having a detector of these samples.

An early idea for detection was to use a second network [38]. However, this network can also be adversarially attacked. More recent statistical approaches include LID [36], which trains the detector on the local intrinsic dimensionality of activations approximated over a batch, and the Mahalanobis detector [33], which trains the detector on the Mahalanobis distances between the activations and a Gaussian fitted to them during training, assuming they are normally distributed. Our detector is not a statistical approach and does not need batch-level statistics, nor statistics from the training data. Detectors trained in the Fourier domain of activations have also been proposed in [20]. See [1] for a review.
Our second contribution is to regularize the network in a way that makes it Holder-continuous, but only on the data distribution's support. Estimations of the Lipschitz constant of a network have been used as estimates of its robustness to adversarial samples in [56; 50; 54; 23], and making the network more Lipschitz (e.g. by penalizing an upper bound on its Lipschitz constant) has been used to make it more robust (i.e. less likely to be fooled) in [23; 11]. These regularizations often work directly on the weights of the network, therefore making it more regular on all the input space. The difference with our method is that we only endue the network with regularity on the support of the clean data. This won't make it more robust to adversarial samples, but it makes its behavior on them more distinguishable, since they tend to lie outside the data manifold.

That adversarial samples lie outside the data manifold, particularly in its co-dimensions, is a common observation and explanation for why adversarial samples are easy to find in high dimensions [18; 52; 49; 36; 46; 29; 2; 16]. To the best of our knowledge, [44] is the only other method that attempts to improve detection

Figure 1: Transformed circles test set from scikit-learn (red and blue) and out-of-distribution points (green) after blocks 6 and 9 of a small ResNet with 9 blocks. In the second row, we add our proposed regularization during training, which makes the movements of the clean points (red and blue) more similar to each other and more different from the movements of the green out-of-distribution points than when using the vanilla network in the first row. In particular, without the regularization, the green points are closer to the clean red points after blocks 6 and 9 which is undesirable.
by encouraging the network during training to learn representations that are more different between clean and adversarial samples. They do this by replacing cross-entropy by a reverse cross-entropy that encourages uniform softmax outputs among the non-predicted classes. We find that our regularization leads to better classification accuracy and adversarial detection than this method.

## 3 Background

Our detector is based on the dynamic viewpoint of neural networks that followed from the analogy between ResNets and the Euler scheme made in [55]. We present this analogy in Section 3.2. The regularization we use was proposed in [26] to improve generalization and we also present it in Section 3.2. The regularity results that follow from this regularization require the use of optimal transport theory, which we present in Section 3.1.

### Optimal Transport

Let \(\alpha\) and \(\beta\) be absolutely continuous densities on a compact set \(\Omega\subset\mathbb{R}^{d}\). The Monge problem is to look for \(T\):\(\mathbb{R}^{d}\)\(\rightarrow\)\(\mathbb{R}^{d}\) moving \(\alpha\) to \(\beta\), i.e. \(T_{\sharp}\alpha\)=\(\beta\), with minimal transport cost:

\[\min_{T\text{ s.t. }T_{\sharp}\alpha=\beta}\int_{\Omega}\|T(x)-x\|_{2}^{2}\, \mathrm{d}\alpha(x) \tag{1}\]

and this problem has a unique solution \(T^{\star}\). An equivalent formulation of the Monge problem in this setting is the dynamical formulation. Here, instead of directly pushing points from \(\alpha\) to \(\beta\) through \(T\), we continuously displace mass from time 0 to 1 according to velocity field \(v_{t}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}\). We denote \(\phi_{t}^{x}\) the position at time \(t\) of the particle that was at \(x\sim\alpha\) at time 0. This position evolves according to \(\partial_{t}\phi_{t}^{x}=v_{t}(\phi_{t}^{x})\). Rewriting the constraint, Problem (1) is equivalent to the dynamical formulation:

\[\min_{v}\int_{0}^{1}\|v_{t}\|_{L^{2}((\phi_{\hat{v}})_{\sharp} \alpha)}^{2}\,\mathrm{d}t \tag{2}\] \[\text{s.t. }\partial_{t}\phi_{t}^{x}=v_{t}(\phi_{t}^{x})\text{ for }x \in\text{support}(\alpha)\text{ and }t\in[0,1[\] \[\phi_{0}=\mathrm{id},(\phi_{1})_{\sharp}\alpha=\beta\]

### Least Action Principle Residual Networks

A residual stage made up of \(M\) residual blocks applies \(x_{m+1}=x_{m}+hr_{m}(x_{m})\) for \(0\leq m<M\), with \(x_{0}\) being the input and \(h\)=1 in practice. The final point \(x_{M}\) is then classified by a linear layer \(F\). The dynamic view considers a residual network as an Euler discretization of a differential equation:

\[x_{m+1}=x_{m}+hr_{m}(x_{m})\;\longleftrightarrow\;\partial_{t}x_{t}=v_{t}( x_{t}) \tag{3}\]
where \(r_{m}\) approximates the vector field \(v_{t}\) at time \(t=m/M\). The dynamic view allows to consider that ResNets are transporting their inputs in space by following a vector field to separate them, the depth representing time, before classification by a linear layer. [26] look for a network \(F\circ T\) that solves the task while having minimal transport cost:

\[\begin{split}\inf_{T,F}&\quad\int_{\Omega}\|T(x)-x \|_{2}^{2}\,\mathrm{d}\alpha(x)\\ \mathrm{s.t.}&\quad\mathcal{L}(F,T_{\sharp}\alpha)=0 \end{split} \tag{4}\]

where \(T\) is made up of the \(M\) residual blocks, \(\alpha\) is the data distribution, \(F\) is the classification head and \(\mathcal{L}(F,T_{\sharp}\alpha)\) is the (cross-entropy) loss obtained from classifying the transformed data distribution \(T_{\sharp}\alpha\) through \(F\). Given Section 3.1, the corresponding dynamical version of (4) is

\[\inf_{v,F} \int_{0}^{1}\|v_{t}\|_{L^{2}((\phi_{\hat{1}})_{\sharp}\alpha)}^{2} \;\mathrm{d}t \tag{5}\] \[\mathrm{s.t.} \partial_{t}\phi_{t}^{x}=v_{t}(\phi_{t}^{x})\text{ for }x\in \mathrm{support}(\alpha)\text{ and }t\in[0,1[\] \[\phi_{0}=\mathrm{id},\;\;\mathcal{L}(F,(\phi_{1})_{\sharp}\alpha)=0\]

[26] show that (4) and (5) are equivalent and have a solution such that \(T\) is an optimal transport map. In practice, (5) is discretized using a sample \(\mathcal{D}\) from \(\alpha\) and an Euler scheme, which gives a residual architecture with residual blocks \(r_{m}\) (parametrized along with the classifier by \(\theta\)) that approximate \(v\). This gives the following problem

\[\min_{\theta} \mathcal{C}(\theta)=\sum_{x\in\mathcal{D}}\;\sum_{m=0}^{M-1}\|r_ {m}(\varphi_{m}^{x})\|_{2}^{2} \tag{6}\] \[\mathrm{s.t} \varphi_{m+1}^{x}=\varphi_{m}^{x}+hr_{m}(\varphi_{m}^{x}),\; \varphi_{0}^{x}=x\;\;\forall\;x\in\mathcal{D}\] \[\mathcal{L}(\theta)=0\]

In practice, we solve Problem (6) using a method of multipliers (see Section 4.2). Our contribution is to show, theoretically and experimentally, that this makes adversarial examples easier to detect.

## 4 Method

We take the view that a ResNet moves its inputs through a discrete vector field to separate them, points in the same class having similar trajectories. Heuristically, for a successful adversarial sample that is close to clean samples, the vector field it follows has to be different at some step from that of the clean samples, so that it joins the trajectory of the points in another class. In Section 4.1, we present how we detect adversarial samples by considering these trajectories. In Section 4.2, we apply the transport regularization by solving (6) to improve detectability of adversarial samples.
### Detection

Given a network that applies \(x_{m+1}=x_{m}+hr_{m}(x_{m})\) to an input \(x_{0}\) for \(0\leq m{<}M\), we consider the embeddings \(x_{m}\) for \(0{<}m\leq M\), or the residues \(r_{m}(x_{m})\) for \(0\leq m{<}M\). To describe their positions in space, we take their norms and their cosine similarities with a fixed vector as features to train our adversarial detector on. Using only the norms already gave good detection accuracy. Cosines to other orthogonal vectors can be added to better locate the points at the price of increasing the number of features. We found that using only one vector already gives state-of-the-art detection, so we only use the norms and cosines to a fixed vector of ones. We train the detector (a random forest in practice, see Section 5.2) on these features. The embeddings \(x_{m}\) and the residues \(r_{m}(x_{m})\) can equivalently describe the trajectory of \(x_{0}\) in space through the blocks. In practice, we use the residues \(r_{m}(x_{m})\), with their norms squared and averaged. So the feature vector given to the random forest for each input \(x_{0}\) that goes through a network that applies \(x_{m+1}=x_{m}+h\ r_{m}(x_{m})\) is

\[\left(\frac{1}{d_{m}}\|r_{m}(x_{m})\|_{2}^{2},\ \cos\Bigl{(}r_{m}(x_{m}), \mathbf{1}_{m}\Bigr{)}\right)_{0\leq m<M} \tag{7}\]

and the label is 0 if \(x_{0}\) is clean and 1 if it is adversarial. Here \(\cos\) is the cosine similarity between two vectors and \(\mathbf{1}_{m}\) is a vector of ones of size \(d_{m}\) where \(d_{m}\) is the size of \(r_{m}(x_{m})\). For any non-residual architecture \(x_{m+1}=g_{m}(x_{m})\), the vector \(x_{m+1}{-}x_{m}\) can be used instead of \(r_{m}(x_{m})\) on layers that have the same input and output dimension, allowing to apply the method to any network with many such layers. And we do test the detector on a ResNeXt, which does not fully satisfy the dynamic view, as the activation is applied after the skip-connection, i.e. \(x_{m+1}=\text{ReLU}(x_{m}+h\ r_{m}(x_{m}))\).

The number of features is twice that of residual blocks (a norm and a cosine per block). This is of the same order as for other popular detectors such as Mahalanobis [33] and LID [36] that extract one feature per residual stage (a residual stage is a group of blocks that keep the same dimension). Even for common large architectures, twice the number of residual blocks is still a small number of features for training a binary classifier (ResNet152 has 50 blocks). More importantly, the features we extract (norms and cosines) are quick to calculate, whereas those of other methods require involved statistical computations on the activations. We include in Appendix 0.D.10 a favorable time comparison of our detector to the Mahalanobis detector. Another advantage is that our detector does not have a hyper-parameter to tune unlike the Mahalanobis and LID detectors.

### Regularization

Regularity of neural networks (typically Lipschitz continuity) has been used as a measure of their robustness to adversarial samples [56, 50, 54, 23, 11]. Indeed, the smaller the Lipschitz constant \(L\) of a function \(f\) satisfying \(\|f(x)-f(y)\|\leq L\|x-y\|\), the less \(f\) changes its output \(f(y)\) for a perturbation (adversarial or
not) \(y\) of \(x\). Regularizing a network to make it more Lipschitz and more robust has therefore been tried in [23] and [11]. For this to work, the regularization has to apply to adversarial points, i.e. outside the support of the clean data distribution. Indeed, the Lipschitz continuity obtained though most of these methods and analyses apply on the entire input space \(\mathbb{R}^{d}\) as they penalize the network's weights directly. Likewise, a small step size \(h\) as in [62] will have the same effect on all inputs, clean or not.

We propose here an alternative approach where we regularize the network only on the support of the input distribution, making it \(\eta\)-Holder on this support (a function \(f\) is \(\eta\)-Holder on \(X\) if \(\forall\ a,b\in X\), we have \(\|f(a)-f(b)\|\leq C\|a-b\|^{\eta}\) for some constants \(C{>}0\) and \(0{<}\eta{\leq}1\), and we denote this \(f\in\mathcal{C}^{0,\eta}(X))\). Since this result does not apply outside the input distribution's support, particularly in the adversarial spaces, then this regularity that only applies to clean samples can serve to make adversarial samples more distinguishable from clean ones, and therefore easier to detect. We show experimentally that the behavior of the network will be more distinguishable between clean and adversarial samples in practice in Section 5.1. We discuss the implementation of the regularization in Section 4.2 and prove the regularity it endues the network with in Section 4.2.

#### 4.2.1 Implementation.

We regularize the trajectory of the samples by solving Problem (6). This means finding, among the networks that solve the task (condition \(\mathcal{L}(\theta)=0\) in (6)), the network that moves the points the least, that is the one with minimal kinetic energy \(\mathcal{C}\). The residual functions \(r_{m}\) we find are then our approximation of the vector field \(v\) that solves the continuous version (5) of Problem (6).

We solve Problem (6) via a method of multipliers: since \(\mathcal{L}\geq 0\), Problem (6) is equivalent to the min-max problem \(\min_{\theta}\max_{\lambda>0}\ \mathcal{C}(\theta)+\lambda\ \mathcal{L}(\theta)\), which we solve, given growth factor \(\tau>0\), and starting from initial weight given to the loss \(\lambda_{0}\) and initial parameters \(\theta_{0}\), through

\[\begin{cases}\theta_{i+1}=\arg\min_{\theta}\ \mathcal{C}(\theta)+\lambda_{i}\ \mathcal{L}(\theta)\\ \lambda_{i+1}=\lambda_{i}+\tau\ \mathcal{L}(\theta_{i+1})\end{cases} \tag{8}\]

We use SGD for \(s{>}0\) steps (i.e. batches) for the minimization in the first line of (8), starting from the previous \(\theta_{i}\). When using a ResNeXt, where a residual block applies \(x_{m+1}=\text{ReLU}(x_{m}+r_{m}(x_{m}))\), we regularize the norms of the true residues \(x_{m+1}{-}x_{m}\) instead of \(r_{m}(x_{m})\).

#### 4.2.2 Theoretical Analysis.

We take \(\Omega{\subset}\mathbb{R}^{d}\) convex and compact and the data distribution \(\alpha{\in}\mathcal{P}(\Omega)\) absolutely continuous such that \(\delta\Omega\) is \(\alpha\)-negligible. We suppose that there exists an open bounded convex set \(X{\subset}\Omega\) such that \(\alpha\) is bounded away from zero and infinity on \(X\) and is zero on \(X^{\complement}\). From [26], Problems (4) and (5) are equivalent and have solutions \((T,F)\) and \((v,F)\) such that \(T\) is an optimal transport map between \(\alpha\) and \(\beta{:=}T_{\sharp}\alpha\). We suppose that \(\beta\) is absolutely continuous and that there exists an open bounded convex set \(Y{\subset}\Omega\) such that \(\beta\)
is bounded away from zero and infinity on \(Y\) and is zero on \(Y^{\complement}\). In the rest of this section, \(v\) solves (5) and we suppose that we find a solution to the discretized problem (6) that is an \(\varepsilon/2\)-approximation of \(v\), i.e. \(\|r_{m}-v_{t_{m}}\|_{\infty}{\leq}\varepsilon/2\) for all \(0{\leq}m{<}M\), with \(t_{m}{=}m/M\).

Definition 1: A function \(f\) is \(\eta\)-Holder on \(X\) if \(\forall\ a,b\in X\), we have \(\|f(a)-f(b)\|\leq C\|a-b\|^{\eta}\) for some constants \(C{>}0\) and \(0{<}\eta{\leq}1\). We denote this \(f\in\mathcal{C}^{0,\eta}(X)\).

In Theorem 3.1, we show that the regularization makes the residual blocks of the network \(\eta\)-Holder (with an error of \(\varepsilon\)) on the support of the input distribution as it moves according to the theoretical vector field solution \(v\). The results hold for all norms on \(\mathbb{R}^{d}\).

Theorem 3.1: _For \(a,b\in\text{support}(\alpha_{t_{m}})\), \(\alpha_{t}{:=}(\phi_{t})_{\sharp}\alpha\) where \(\phi\) solves (5) along with \(v\), we have_

\[\|r_{m}(a)-r_{m}(b)\|\leq\varepsilon+K\|a-b\|^{\zeta_{1}}\text{ if }\|a-b\|\leq 1\] \[\|r_{m}(a)-r_{m}(b)\|\leq\varepsilon+K\|a-b\|^{\zeta_{2}}\text{ if }\|a-b\|>1\]

_for constants \(K>0\) and \(0<\zeta_{1}\leq\zeta_{2}\leq 1\)._

Proof: The detailed proof is in Appendix 0.C.1. First, we have that \(v_{t}=(T-\texttt{id})\circ T_{t}^{-1}\) where \(T_{t}:=(1-t)\texttt{id}+tT\) and \(T\) solves (4). Being an optimal transport map, \(T\) is \(\eta\)-Holder. So for all \(a,b\in\text{support}(\alpha_{t})\) and \(t\in[0,1[\), where \(\alpha_{t}=(\phi_{t}^{\cdot})_{\sharp}\alpha=(T_{t})_{\sharp}\alpha\) with \(\phi\) solving (5) with \(v\), we have

\[\|v_{t}(a)-v_{t}(b)\|\leq\|T_{t}^{-1}(a)-T_{t}^{-1}(b)\|+C\|T_{t}^{-1}(a)-T_{t} ^{-1}(b)\|^{\eta} \tag{9}\]

We then show that \(T_{t}^{-1}\) is an optimal transport map and so is \(\eta_{t}\)-Holder with \(0{<}\eta_{t}{\leq}1\). Using the hypothesis on \(r\) and the triangle inequality, we get, for all \(a,b\in\text{support}(\alpha_{t_{m}})\)

\[\|r_{m}(a)-r_{m}(b)\|\leq\ \varepsilon+C_{t_{m}}\|a-b\|^{\eta_{t_{m}}}+CC_{t_{m}} ^{\eta}\|a-b\|^{\eta\eta_{t_{m}}} \tag{10}\]

Then set the constants \(K\), \(\zeta_{1}\) and \(\zeta_{2}\) as necessary.

We use Theorem 3.1 to now bound the distance between the residues at depth \(m\) as a function of the distance between the network's inputs. For inputs \(a_{0}\) and \(b_{0}\) to the network, the intermediate embeddings are \(a_{m+1}=a_{m}+hr_{m}(a_{m})\) and \(b_{m+1}=b_{m}+hr_{m}(b_{m})\), and the residues used to compute features for adversarial detection are \(r_{m}(a_{m})\) and \(r_{m}(b_{m})\). So we want to bound \(\|r_{m}(a_{m})-r_{m}(b_{m})\|\) as a function of \(\|a_{0}-b_{0}\|\). This is usually done by multiplying the Lipschitz constants of each block up to depth \(m\), which leads to an overestimation [24], or through more complex estimation algorithms [54, 32, 6]. Bound (9) allows through \(T_{t}^{-1}\) to avoid multiplying the Holder constants of the blocks. If \(a_{0}\) and \(b_{0}\) are on the clean data support \(X\), we get Theorem 3.1 below with proof in Appendix 0.C.2.
Theorem 4.1: _For \(a_{0},b_{0}\in X\) and constants \(C,L{>}0\),_

\[\|r_{m}(a_{m})-r_{m}(b_{m})\| \leq \ \varepsilon+\|a_{0}-b_{0}\|+C\|a_{0}-b_{0}\|^{\eta}+\] \[+ L(\|a_{m}-\phi_{t_{m}}^{a_{0}}\|+\|b_{m}-\phi_{t_{m}}^{b_{0}}\|)\]

Term \(\mu(a_{0}){:=}\|a_{m}{-}\phi_{t_{m}}^{a_{0}}\|\) (and \(\mu(b_{0}){:=}\|b_{m}{-}\phi_{t_{m}}^{b_{0}}\|\)) is the distance between the point \(a_{m}\) after \(m\) residual blocks and the point \(\phi_{t_{m}}^{a_{0}}\) we get by following the theoretical solution vector field \(v\) up to time \(t_{m}\) starting from \(a_{0}\). If \(a_{0}\) and \(b_{0}\) are not on the data support \(X\), an extra term has to be introduced to use bound (9). Bounding the terms \(\mu(a_{0})\) and \(\mu(b_{0})\) is possible under more regularity assumptions on \(v\). We assume then that \(v\) is \(\mathcal{C}^{1}\) and Lipschitz in \(x\), which is not stronger than the regularity we get on \(v\) through our regularization, as it does not give a similar result to bound (9). We have for all inputs \(a_{0}\) and \(b_{0}\), whether they are clean or not, Theorem 4.1 below with proof in Appendix 0.C.2.

Theorem 4.2: _For \(a_{0},b_{0}\in\mathbb{R}^{d}\) and constants \(R,S>0\),_

\[\|r_{m}(a_{m})-r_{m}(b_{m})\| \leq \ \varepsilon+LS\varepsilon+LSRh+\|a_{0}-b_{0}\|+C\|a_{0}-b_{0}\|^{ \eta}+\] \[+ LS(\mathit{dist}(a_{0},X)+\mathit{dist}(b_{0},X))\]

Terms \(\mathrm{dist}(a_{0},X)\) and \(\mathrm{dist}(b_{0},X)\) show that the regularity guarantee is increased for inputs in \(X\). The trajectories of clean points are then closer to each other and more different from those of abnormal samples outside \(X\).

## 5 Experiments

We evaluate our method on adversarial samples found by 8 attacks. The threat model is as follows. We use 6 white-box attacks that can access the network and its weights and architecture but not its training data: FGM [19], BIM [31], DF [40], CW [8], AutoAttack (AA) [13] and the Auto-PGD-CE (APGD) variant of PGD [37], and 2 black-box attacks that only query the network: HSJ [10] and BA [7]. We assume the attacker has no knowledge of the detector and use the untargeted (i.e. not trying to direct the mistake towards a particular class) versions of the attacks. We use a maximal perturbation of \(\epsilon{=}0.03\) for FGM, APGD, BIM and AA. We use the \(L_{2}\) norm for CW and HSJ and \(L_{\infty}\) for the other attacks. We compare our detector (which we call the Transport detector or TR) to the Mahalanobis detector (MH in the tables below) of [33] and to the detector of [28, 27] that uses natural scene statistics (NS in the tables below), and our regularization to reverse cross entropy training of [44], which is also meant to improve detection of adversarial samples. We use ART [43] and its default hyper-parameter values (except those specified) to generate the adversarial samples, except for AA for which we use the authors' original code. The code is available at github.com/skander-karkar/adv. See Appendix 0.D.1 for more details.

We use 3 networks and datasets: ResNeXt50 on CIFAR100, ResNet110 on CIFAR10 and WideResNet on TinyImageNet. Each network is trained normally
with cross entropy, with the transport regularization added to cross entropy (called a LAP-network for Least Action Principle), and with reverse cross entropy instead of cross entropy (called an RCE-network). For LAP training, we use (8) with \(\tau\)=1, \(s\)=1 and \(\lambda_{0}\)=1 for all networks. These hyper-parameters are chosen to improve validation accuracy during training not adversarial detection. Training details are in Appendix 0.D.2.

In Section 5.1, we conduct preliminary experiments to show that LAP training improves generalization and stability, and increases the difference between the transport costs of clean and adversarial samples. In Section 5.2, we test our detector when it is trained and tested on samples generated by the same attack. In Section 5.3, we test our detector when it is trained on samples generated by FGM and tested on samples from the other attacks. We then consider OOD detection and adaptive attacks on the detector.

### Preliminary Experiments

Our results confirm those in [26] that show that LAP training improves test accuracy. Vanilla ResNeXt50 has an accuracy of 74.38% on CIFAR100, while LAP-ResNeXt50 has an accuracy of 77.2%. Vanilla ResNet110 has an accuracy of 92.52% on CIFAR10, while LAP-ResNet110 has an accuracy of 93.52% and the RCE-ResNet110 of 93.1%. Vanilla WideResNet has an accuracy of 65.14% on TinyImageNet, while LAP-WideResNet has an accuracy of 65.34%. LAP training is also more stable by allowing to train deep networks without batch-normalization in Figure 4 in Appendix 0.D.4.

We see in Figure 2 that LAP training makes the transport cost \(\mathcal{C}\) more different between clean and adversarial points. Using its empirical quantiles on clean points allows then to detect samples from some attacks with high recall and a fixed false positive rate, without seeing adversarial samples.

Figure 2: Histogram of transport cost \(\mathcal{C}\) for clean and FGM-attacked test samples with different values of \(\epsilon\) on CIFAR100. The vertical lines represent the 0.02 and 0.98 empirical quantiles of the transport cost of the clean samples. Left: ResNeXt50. Right: LAP-ResNeXt50.
### Detection of Seen Attacks

For detection training, the test set is split in 0.9/0.1 proportions into two datasets, B1 and B2. For each image in B1 (respectively B2), an adversarial sample is generated and a balanced detection training set (respectively a detection test set) is created. Since adversarial samples are created for a specific network, this is done for the vanilla version of the network and its LAP and RCE versions. We tried augmenting the detection training dataset with a randomly perturbed version of each image, to be considered clean during detection training, as in [33], but we found that this does not improve detection accuracy. This dataset creation protocol is standard and is depicted in Figure 3 in Appendix D.3. We did not limit the datasets to successfully attacked images only as in [33], as we consider the setting of detecting all adversarial samples, whether or not they fool the network, more challenging (which is seen in the results). It also allows to detect any attempted interference with the network, even if it fails at fooling it.

Samples in the detection training set are fed through the network and the features for each detector are extracted. We tried three classifiers (logistic regression, random forest and SVM) trained on these features for all detectors, and kept the random forest as it always performs best. We tried two methods to improve the accuracy of all detectors: class-conditioning and ensembling. In class-conditioning, the features are grouped by the class predicted by the network, and a detector is trained for every class. At test time, the detector trained on the features of the predicted class is used. A detector is also trained on all samples regardless of the predicted class and is used in case a certain class is never targeted by the attack. We also tried ensembling the class-conditional detector with the general all-class detector: an input is considered an attack if at least one detector says so. This ensemble of the class-conditional detector and the general detector performs best for all detectors, and is the one we use.

We report the accuracy of each detector on the detection test set for both the vanilla and the LAP network in Table 1. In each cell, the first number corresponds to the vanilla network and the second to the regularized LAP-network. Since the NS detector takes the image and not its embeddings as input, the impact of LAP and RCE training on its performance is minimal and we report its performance on the vanilla network only. These results are averaged over 5 runs and the standard deviations (which are tight) are in Tables 3 to 7 in Appendix D.5, along with results on RCE-networks. Since some attacks are slow, we don't test them on all network-dataset pairs in this experiment. Results in Table 1 show two things. First, our detector performs better than both other detectors, with or without the regularization. Second, both the TR and MH detectors work better on the LAP-networks most times. The MH detector benefits more from the regularization, but on all attacks, the best detector is always the Transport detector. In the tables in Appendix D.5, RCE often improves detection accuracy in this experiment, but clearly less than LAP training. On CIFAR10, our detector outperforms the MH detector by 9 to 16 percentage points on the vanilla ResNet110, and the NS detector by up to 5 points. LAP training improves the accuracy of our detector by an average 1.5 points and that of the MH detector by a substantial
8.3 points on average. On CIFAR100, our detector outperforms the MH detector by 1 to 5 points on the vanilla ResNeXt50, and the NS detector by up to 3 points. LAP training improves the accuracy of both detectors by an average 1 point. On TinyImageNet, our detector greatly outperforms the MH detector by 3 to 15 points on the vanilla WideResNet, and the NS detector slightly. LAP training does not change the accuracy of our detector and improves that of the MH detector by 0.85 points on average. Detection rates of successful adversarial samples (i.e. those that fool the network) are in Table 14 in Appendix D.7 and are higher than 95% on our detector. False positive rates (positive meaning adversarial) are in Table 16 in Appendix D.8 and are always less than 5% on our detector. The AUROC is in Table 18 in Appendix D.9. On all these metrics, our detector outperforms the other detectors largely, and LAP-training greatly improves the performance of the Mahalanobis detector.

\begin{table}
\begin{tabular}{l c c c c} \hline \multirow{2}{*}{Attack} & \multirow{2}{*}{Detector} & ResNet110 & ResNeXt50 & WideResNet \\  & & CIFAR10 & CIFAR100 & TinyImageNet \\ \hline \multirow{4}{*}{FGM} & TR & 97.14/**98.70** & 97.26/**98.32** & **95.36/**95.14** \\  & MH & 87.78/95.64 & 95.82/96.82 & 81.06/85.26 \\  & NS & 94.56 & 94.70 & 94.90 \\ \hline \multirow{4}{*}{APGD} & TR & 94.10/**97.50** & 96.04/**97.84** & **95.22/**95.20** \\  & MH & 82.08/90.70 & 93.94/94.60 & 79.66/85.10 \\  & NS & 94.28 & 94.18 & 94.86 \\ \hline \multirow{4}{*}{BIM} & TR & 97.54/**99.28** & 98.02/**98.92** & **95.26/**95.12** \\  & MH & 86.78/95.38 & 96.06/97.76 & 81.20/82.46 \\  & NS & 95.04 & 94.72 & 95.00 \\ \hline \multirow{4}{*}{AA} & TR & 88.88/**94.08** & 84.90/**87.56** & **81.38/**81.24** \\  & MH & 80.46/89.96 & 83.90/86.58 & 78.40/78.40 \\  & NS & 88.78 & 84.82 & 81.32 \\ \hline \multirow{4}{*}{DF} & TR & **99.98**/99.84 & **99.80**/99.58 & \multirow{4}{*}{} \\  & MH & 91.50/96.70 & 97.30/97.12 & \\  & NS & 99.78 & 99.6 & \\ \hline \multirow{4}{*}{CW} & TR & **98.04**/97.96 & 97.04/**97.80** & \multirow{4}{*}{} \\  & MH & 85.58/93.36 & 95.38/96.42 & \\  & NS & 93.86 & 90.7 & \\ \hline \multirow{4}{*}{HSJ} & TR & **99.94**/99.92 & \multirow{4}{*}{} \\  & MH & 85.50/94.56 & \multirow{4}{*}{} \\  & NS & 99.68 & \\ \hline \multirow{4}{*}{BA} & TR & 96.56/**97.02** & \multirow{4}{*}{} \\  & MH & 80.20/89.62 & \\ \cline{1-1}  & NS & 92.10 & \\ \hline \end{tabular}
\end{table}
Table 1: Average accuracy of detectors on adversarial samples from seen attacks on Network/LAP-Network over 5 runs.
### Detection of Unseen Attacks

An important setting is when we don't know which attack might be used or only have time to train detectors on samples from one attack. We still want our detector to generalize well to unseen attacks. To test this, we use the same vanilla networks as above but the detectors are now trained on the detection training set created by the simplest and quickest attack (FGM) and tested on the detection test sets created by the other attacks. Results are in Table 2. We see that our detector has very good generalization to unseen attacks, even those very different from FGM, comfortably better than the MH detector, by up to 19 percentage points, while the NS detector only generalizes to variants of FGM (APGD and BIM), and fails on the other attacks. These results are averaged over 5 runs and the standard deviations are in Tables 8 to 13 in Appendix D.6. On our detector, the detection rate of successful adversarial samples remains higher than 90% in most cases (Table 15 in Appendix D.7) and the FPR is always lower than 10% (Table 17 in Appendix D.8). The AUROC is in Table 19 in Appendix D.9. Our detector almost always outperforms the other detectors on all these metrics.

\begin{table}
\begin{tabular}{l c c c c} Attack & Detector & \begin{tabular}{c} ResNet110 \\ CIFAR10 \\ \end{tabular} & \begin{tabular}{c} ResNeXt50 \\ CIFAR100 \\ \end{tabular} & 
\begin{tabular}{c} WideResNet \\ TinyImageNet \\ \end{tabular} \\ \hline \multirow{3}{*}{APGD} & TR & 89.32 & 91.94 & 93.26 \\  & MH & 77.34 & 90.86 & 76.96 \\  & NS & **92.08** & **92.16** & **94.06** \\ \hline \multirow{3}{*}{BIM} & TR & **96.02** & **95.02** & **94.66** \\  & MH & 77.24 & 93.16 & 77.02 \\  & NS & 93.88 & 93.88 & 94.62 \\ \hline \multirow{3}{*}{AA} & TR & **85.10** & **73.32** & **77.04** \\  & MH & 72.12 & 73.08 & 60.36 \\  & NS & 51.82 & 51.32 & 65.60 \\ \hline \multirow{3}{*}{DF} & TR & **91.02** & **85.16** & **90.62** \\  & MH & 80.12 & 82.72 & 73.18 \\  & NS & 51.40 & 51.62 & 72.82 \\ \hline \multirow{3}{*}{CW} & TR & **93.18** & **78.18** & **91.42** \\  & MH & 79.92 & 76.44 & 75.52 \\  & NS & 50.84 & 51.02 & 71.96 \\ \hline \multirow{3}{*}{HSJ} & TR & **93.00** & **85.04** & \\  & MH & 79.70 & 82.82 & \\  & NS & 52.12 & 52.04 & \\ \hline \multirow{3}{*}{BA} & TR & **90.92** & **92.14** & \\  & MH & 79.32 & 84.46 & \\ \cline{1-1}  & NS & 59.88 & 57.90 & \\ \hline \end{tabular}
\end{table}
Table 2: Average accuracy of detectors on samples from unseen attacks after training on FGM over 5 runs.
However, this experiment shows that our regularization has some limitations. We see in Tables 8 to 13 in Appendix D.6 that LAP training does not improve detection accuracy as much, and sometimes reduces it. It still improves it for the MH detector on all attacks on ResNet110 and WideResNet by up to 10 points, and LAP training still always does better than RCE training. We claim this is because these methods reduce the variance of features extracted on the seen attack, harming generalization to unseen attacks. This explains why detection of APGD and BIM, variants of FGM, improves.

### Detection of Out-Of-Distribution Samples

Since our analysis applies to all out-of-distribution (OOD) samples, we test detection of OOD samples in a similar setting to [33]. We train a model on a first dataset (ResNet110 on CIFAR10 and ResNeXt50 on CIFAR100), then train detectors to tell this first dataset from a second dataset (which can be an adversarially attacked version of the first dataset), then test their ability to tell the first dataset from a third unseen dataset (SVHN). Our detector does very well and better than the MH detector on both experiments, and detection accuracy of samples from the unseen distribution is higher than 90% when using the CW attack to create the second dataset. Details are in Appendix D.11.

### Attacking the Detector

We consider the case where the detector is also attacked (adaptive attacks). We try 2 attacks on the TR and MH detectors. Both are white-box with respect to the network. The first is black-box with respect to the detector and only knows if a sample has been detected or not. The second has some knowledge about the detector. It knows what features it uses and can attack it directly to find adversarial features. We test these attacks by looking at the percentage of detected successful adversarial samples that they turn into undetected successful adversarial samples. For the first attack, this is 6.8% for our detector and 12.9% for the MH detector on the LAP-ResNet110, and is lowered by LAP training. For the second attack it is 14% on our detector. Given that detection rates of successful adversarial samples are almost 100% (see Appendix D.7), this shows that an adaptive attack does not circumvent the detector, as detection rates drop to 85% at worst. Details are in Appendix D.12.

## 6 Conclusion

We proposed a method for detecting adversarial samples, based on the dynamical view of neural networks. The method examines the discrete vector field moving the inputs to distinguish clean and abnormal samples. The detector requires minimal computation to extract the features it uses for detection and achieves state-of-the-art detection accuracy on seen and unseen attacks. We also use a transport regularization that both improves test classification accuracy and the accuracy of adversarial detectors.
## Ethical Statement

Adversarial detection and robustness are essential to safely deploy neural networks that attackers might target for nefarious purposes. But adversarial attacks can be used to evade neural networks that are deployed for nefarious purposes.
