# Optimization for Robustness Evaluation beyond $\ell_{p}$ Metrics

###### Abstract

Empirical evaluation of deep learning models against adversarial attacks
entails solving nontrivial constrained optimization problems. Popular
algorithms for solving these constrained problems rely on projected gradient
descent (PGD) and require careful tuning of multiple hyperparameters.
Moreover, PGD can only handle $\ell_{1}$, $\ell_{2}$, and $\ell_{\infty}$
attack models due to the use of analytical projectors. In this paper, we
introduce a novel algorithmic framework that blends a general-purpose
constrained-optimization solver PyGRANSO, With Constraint-Folding (PWCF), to
add reliability and generality to robustness evaluation. PWCF 1) finds good-
quality solutions without the need of delicate hyperparameter tuning, and 2)
can handle general attack models, e.g., general $\ell_{p}$ ($p\geq 0$) and
perceptual attacks, which are inaccessible to PGD-based algorithms.

Index Terms—  deep neural networks, adversarial robustness, adversarial
attack, robustness evaluation, perceptual metrics, constrained optimization.

##  1 Introduction

In visual recognition, deep neural networks (DNNs) are not robust against
perturbations that are easily discounted by human perception—either
adversarially constructed or naturally occurring [1, 2, 3, 4, 5, 6, 7, 8, 9].
A popular way of finding an adversarial perturbation (a.k.a adversarial
attack) is by solving the _adversarial loss_ formulation [10]:

$\displaystyle\begin{split}&\max_{\mathbf{x}^{\prime}}\ell\pqty{\mathbf{y},f_{\mathbf{\theta}}(\mathbf{x}^{\prime})}\\\
\operatorname{s.t.}\;\mathbf{x}^{\prime}\in\Delta(\mathbf{x})&=\\{\mathbf{x}^{\prime}\in[0,1]^{n}:d\pqty{\mathbf{x},\mathbf{x}^{\prime}}\leq\varepsilon\\}\end{split}.$
(1)

Here, $f_{\theta}$ is the DNN model, and $\Delta(\mathbf{x})$ is an allowable
perturbation set with radius $\varepsilon$ as measured by the metric $d$.
Early works assume $\Delta(\mathbf{x})$ is the $\ell_{p}$ norm ball
intersected with the natural image box, i.e.,
$\\{\mathbf{x}^{\prime}\in[0,1]^{n}:\norm{\mathbf{x}-\mathbf{x}^{\prime}}_{p}\leq\varepsilon\\}$,
where $p=1,2,\infty$ are popular choices [10, 2]. To capture visually
realistic perturbations, recent works have also modeled nontrivial
transformations using non-$\ell_{p}$ metrics [3, 4, 5, 6, 7, 8, 9, 11]. As for
empirical robustness evaluation (RE), solutions of Eq. 1 lead to the worst-
case perturbations to fool $f_{\mathbf{\theta}}$.

But solving Eq. 1 is not easy: the objective is non-concave for typical
choices of loss $\ell$ and model $f_{\mathbf{\theta}}$; for non-$\ell_{p}$
metrics, $\Delta(\mathbf{x})$ is often a complicated nonconvex set. In
practice, there are two major lines of algorithms: (a) direct numerical
maximization that takes differentiable $\ell$ and $f_{\mathbf{\theta}}$, and
tries direct maximization, e.g., using gradient-based methods [10, 12]. This
often only produces a suboptimal solution and can lead to overoptimistic RE;
(b) upper-bound maximization that constructs tractable upper bounds for the
margin loss $\ell_{\mathrm{ML}}=\max_{i\neq
y}f_{\mathbf{\theta}}^{i}(\mathbf{x}^{\prime})-f_{\mathbf{\theta}}^{y}(\mathbf{x}^{\prime})$,
where $y$ is the true class of $\mathbf{x}$ and the boldface $\mathbf{y}$ is
the one-hot encoding of $y$, and then optimizes against the upper bounds [13].
Improving the tightness of upper bounding while maintaining tractability is
still an active area of research.

Another formalism of robustness is the _robustness radius_ (or minimum
distortion radius), defined as the minimal level of perturbation that causes
$f_{\mathbf{\theta}}$ to change its predicted class:

$\displaystyle\min_{\mathbf{x}^{\prime}\in[0,1]^{n}}\;d\pqty{\mathbf{x},\mathbf{x}^{\prime}}\quad\operatorname{s.t.}\;\max_{i\neq
y}f_{\mathbf{\theta}}^{i}(\mathbf{x}^{\prime})\geq
f_{\mathbf{\theta}}^{y}(\mathbf{x}^{\prime}).$ (2)

Solving Eq. 2 produces not only a minimal perturbation $\mathbf{x}^{\prime}$,
but also a robustness radius, making Eq. 2 another popular choice for RE [14,
12, 15]. In fact, [14, 16, 15] perform adversarial attacks by trying to solve
Eq. 2.

In this paper, we focus on numerical optimization of Eq. 1. In particular, we
(I) adapt the constrained-optimization solver PyGRANSO [17, 18] with a
constraint-folding (PWCF) technique—crucial for making PyGRANSO solve Eq. 1
with reasonable speed and quality, and (II) show that PWCF can handle _attacks
other than the popular but unrealistic $\ell_{1}$, $\ell_{2}$, and
$\ell_{\infty}$ ones_—beyond the reach of PGD-based methods. This can lead to
considerably improved RE as PWCF (I) can serve as a _reliable supplement_ to
the state-of-the-art (SOTA) RE packages on $\ell_{1}$, $\ell_{2}$, and
$\ell_{\infty}$ attacks, e.g. AutoAttack [12], and (II) opens up the
possibility of RE over a much wider range of attack models, e.g., general
$\ell_{p}$ attacks with any $p>0$ and more complicated ones such as perceptual
attacks [11]. We remark that PWCF is also general enough to solve Eq. 2, but
due to the limited preliminary results currently at hand, we leave it as
future work.

##  2 Technical background

Eq. 1 is often solved by the projected gradient descent (PGD)111It should be
“ascent” instead of “descent” due to the maximization, but we follow the
AutoAttack package. method. The basic update reads
$\mathbf{x}^{\prime}_{new}=\mathcal{P}_{\Delta(\mathbf{x})}(\mathbf{x}^{\prime}_{old}+t\nabla\ell(\mathbf{x}^{\prime}_{old}))$,
where $\mathcal{P}_{\Delta(\mathbf{x})}$ is the projection operator onto
$\Delta(\mathbf{x})$. When
$\Delta(\mathbf{x})=\\{\mathbf{x}^{\prime}\in[0,1]^{n}:\norm{\mathbf{x}^{\prime}-\mathbf{x}}_{p}\leq\varepsilon\\}$
with $p=1,\infty$, $\mathcal{P}_{\Delta(\mathbf{x})}$ takes simple forms. For
$p=2$, sequential projection onto the box and then the norm ball at least
finds a feasible solution. Hence, PGD is feasible for these cases. _For other
choices of $p$ and general non-$\ell_{p}$ metrics $d$ where analytical
projection is hard to derive, PGD does not apply._ For practical PGD methods,
previous works have shown that the solution quality is sensitive to the tuning
of multiple hyperparameters, e.g., step-size schedule and iteration budget
[19, 20, 12]. The SOTA PGD variants, APGD-CE and APGD-DLR, try to make the
tuning automatic by combining a heuristic adaptive step-size schedule and
momentum acceleration under fixed iteration budget [12]—both are built into
the popular AutoAttack package222https://github.com/fra31/auto-attack.

###  2.1 PyGRANSO for constrained optimization

In principle, as an instance of general nonlinear optimization (NO) problems
[21]

$\displaystyle\begin{split}&\min_{\mathbf{x}}\;g(\mathbf{x})\\\
\operatorname{s.t.}\;c_{i}(\mathbf{x})\leq
0\;\forall\;&i\in\mathcal{I};\;h_{j}(\mathbf{x})=0\;\forall\;j\in\mathcal{E},\end{split}$
(3)

Eq. 1 can be solved by general-purpose NO solvers such as Knitro [22], Ipopt
[23], and GENO [24]. However, there are two caveats: (1) the above solvers
only handle continuously differentiable objective and constraint functions,
i.e., $g$, $c_{i}$’s, and $h_{j}$’s, but non-differentiable $g$, $c_{i}$’s,
and $h_{j}$’s are common in Eq. 1, e.g., when $d$ is the $\ell_{1}$ or
$\ell_{\infty}$ distance, or $f_{\mathbf{\theta}}$ uses non-differentiable
activations; (2) they require analytical gradients of $g$, $c_{i}$’s, and
$h_{j}$’s, which are impractical to derive when DNN models
$f_{\mathbf{\theta}}$ are involved.

PyGRANSO333https://ncvx.org [18, 17] is a recent PyTorch-port of the powerful
MATLAB package GRANSO [17] which can handle general NO problems of form Eq. 3
with non-differentiable $g$, $c_{i}$’s, and $h_{j}$’s. It only requires these
functions to be _almost everywhere differentiable_ , which are satisfied by
almost all forms of Eq. 1 proposed so far in the literature. GRANSO employs a
quasi-Newton sequential quadratic programming (BFGS-SQP) to solve Eq. 3, and
features a rigorous adaptive step-size rule via line search and a principled
stopping criterion inspired by gradient sampling [25]. PyGRANSO equips GRANSO
with auto-differentiation and GPU computing powered by PyTorch—crucial for
deep learning problems. The stopping criterion is controlled by stationarity,
total constraint violation, and optimization tolerance—all can be
transparently controlled, but is typically unnecessary to tune.

##  3 PyGRANSO with constraint folding as a generic solver for Eq. 1

Though PyGRANSO can serve as a promising solver for Eq. 1 with general metric
$d$, we find in practice that naive deployment can suffer from slow
convergence, or low quality solutions due to numerical issues. Below, we
introduce PyGRANSO With Constraint-Folding (PWCF) and other problem-specific
techniques, which can substantially speed up the optimization process and
improve the solution quality.

###  3.1 Reformulating the $\ell_{\infty}$ constraint to avoid sparse
subgradients

The BFGS-SQP algorithm inside PyGRANSO relies on the subgradients of the
objective and the constraint functions to approximate the (inverse) Hessian
and to compute the search direction. Hence, when the subgradients are sparse,
updating all optimization variables may take many iterations, leading to slow
convergence. For the $\ell_{\infty}$ metric,

$\displaystyle\partial_{\mathbf{z}}\norm{\mathbf{z}}_{\infty}=\operatorname{conv}\\{\mathbf{e}_{k}\operatorname{sign}(z_{k}):z_{k}=\norm{\mathbf{z}}_{\infty}\;\forall\,k\\},$
(4)

where $\mathbf{e}_{k}$’s are the standard basis vectors, $\operatorname{conv}$
denotes convex hull, and
$\operatorname{sign}(z_{k})=z_{k}/\absolutevalue{z_{k}}$ if $z_{k}\neq 0$,
else $[-1,1]$. The subgradient in Eq. 4 contains no more than
$\absolutevalue{\\{k:z_{k}=\norm{\mathbf{z}}_{\infty}\\}}$ nonzeros, and hence
is sparse when $\absolutevalue{\\{k:z_{k}=\norm{\mathbf{z}}_{\infty}\\}}$ is
small. To avoid this issue, we propose a reformulation

$\norm{\mathbf{x}-\mathbf{x}^{\prime}}_{\infty}\leq\varepsilon\Longleftrightarrow-\varepsilon\mathbf{1}\leq\mathbf{x}-\mathbf{x}^{\prime}\leq\varepsilon\mathbf{1},$
(5)

where $\mathbf{1}\in\mathbb{R}^{n}$ is the all-one vector.

###  3.2 Constraint-folding to reduce the number of constraints

Fig. 1: Optimization trajectory of the objective value and constraint
violation w.r.t iterations for an $\ell_{\infty}$ case on CIFAR-10 dataset.
woR: using $\ell_{\infty}$ original form; wR: with reformulation but no
folding; wRF: with reformulation and folding. Maximum time budget per curve:
$600s$ (only wRF terminates before reaching this budget). Both objective and
violation reaching $0$ indicates successful attack.

The natural image constraint $\mathbf{x}^{\prime}\in[0,1]^{n}$ is a set of $n$
box constraints. The reformulation described in Section 3.1 introduces another
$\Theta(n)$ box constraints. Although all these are just simple linear
constraints, the $\Theta(n)$-growth is daunting: for natural images, $n$ is
the number of pixels that can easily get into hundreds of thousands. Typical
NO problems become more difficult the number of constraints grows, e.g.,
leading to slow convergence for numerical algorithms.

To combat this, we introduce a folding technique that can reduce the number of
constraints into a small constant. To see how this is possible, first note
that any equality constraint $h(\mathbf{x})=0$ or inequality constraint
$c(\mathbf{x})\leq 0$ can be reformulated as

$\displaystyle
h(\mathbf{x})=0\Longleftrightarrow\absolutevalue{h(\mathbf{x})}\leq
0~{}\text{,}$ (6) $\displaystyle c(\mathbf{x})\leq
0\Longleftrightarrow\max\\{c(\mathbf{x}),0\\}\leq 0.$ (7)

We can then fold them together as

$\mathcal{F}(|h(\mathbf{x})|,\max\\{c(\mathbf{x}),0\\})\leq 0,$ (8)

where $\mathcal{F}:\mathbb{R}_{+}^{2}\mapsto\mathbb{R}_{+}$
($\mathbb{R}_{+}\doteq\\{t:t\geq 0\\}$) can be any function satisfying
$\mathcal{F}(\mathbf{z})=0\Longrightarrow\mathbf{z}=\mathbf{0}$, e.g., any
$\ell_{p}$ ($p\geq 1$) norm.

It is easy to verify the equivalence of Eq. 8 and the original constraints in
Eq. 6. The folding technique can be used to a subset or all of the
constraints; one can group and then fold constraints according to their
physical meanings. We note that folding or aggregating constraints is not a
new idea and has been popular in engineering design. For example, [26] uses
$\ell_{\infty}$ folding and its log-sum-exponential approximation to deal with
numerous design constraints. However, applying folding into NO problems in
machine learning seems rare, potentially because producing non-differentiable
constraint(s) due to the folding seems counterproductive.

In our experiments, we use $\mathcal{F}=\norm{\cdot}_{2}$ to fold the
$\Theta(n)$ box constraints from $\ell_{\infty}$ reformulation into a single
constraint, enforce the $\mathbf{x}^{\prime}\in[0,1]^{n}$ constraints in
$f_{\mathbf{\theta}}$ by direct clipping. Fig. 1 shows clearly that combining
folding and reformulation can substantially speed up convergence and boost the
solution quality for our algorithm.

###  3.3 Loss clipping when solving Eq. 1 with PWCF

For Eq. 1 with the popular cross-entropy (CE) and margin losses, the objective
value can easily dominate constraint violation during the maximization
process. Since PyGRANSO tries to balance the objective value and constraint
violation when making progress, it can persistently prioritize optimizing the
objective over constraint satisfaction, resulting in very slow progress in
finding a feasible solution. To resolve this numerical difficulty, we propose
using clipped margin loss $\ell_{ML}$ with maximal value $0.01$, as any
$\ell_{ML}\geq 0$ indicates a successful attack. For the same reason, we use
clipped CE loss with maximal value at $10$ in PWCF444Attack success happens
when the true logit output less than $1/K$ (assuming softmax normalization is
applied), where $K$ is the number of classes. So the critical value is $-\log
1/K$, which is $<10$ for $K\leq e^{10}$, sufficient for typical RE datasets. .

##  4 Experiments: solving Eq. 1 with PWCF

###  4.1 PWCF offers competitive and complementary attack performance to Eq. 1

Table 1: Comparison of our PWCF with SOTA attack methods on $\ell_{1}$-, $\ell_{2}$\- and $\ell_{\infty}$\- attacks. For given pretrained models, we report the models’ clean and robust accuracy—lower robust accuracy means more effective attacks. We test on both CE and margin loss for APGD and PWCF. Numbers are in $(\%)$. Model - Attack denotes the selected combination of pretrained model and attack type/radius.  |  |  | APGD | PWCF(ours) | Square | APGD
---|---|---|---|---|---|---
Dataset | Model - Attack | Clean | CE | M | CE | M | M | +PWCF
$\text{CIFAR}10$ | P1 [27] \- $\ell_{1}(12)$ | $73.3$ | $0.96$ | $0.00$ | $28.6$ | $0.00$ | $2.28$ | 0.00
| WRN-70-16 [28] \- $\ell_{2}(0.5)$ | $94.7$ | $81.8$ | $81.1$ | $81.8$ | $81.0$ | $87.9$ | 80.8
| WRN-70-16 [28] \- $\ell_{\infty}(0.03)$ | $90.8$ | $69.4$ | $68.0$ | $73.6$ | $72.8$ | $71.6$ | 67.1
$\text{ImageNet}100$ | PAT-Alex [11] \- $\ell_{2}(4.7)$ | $75.0$ | $42.7$ | $44.0$ | $42.8$ | $44.5$ | $63.1$ | 40.9
| PAT-Alex [11] \- $\ell_{\infty}(0.016)$ | $75.0$ | $48.0$ | $48.2$ | $56.6$ | $48.8$ | $59.9$ | 45.2

We take SOTA $\ell_{1}$-, $\ell_{2}$-, and $\ell_{\infty}$ adversarially-
trained models on
CIFAR10555https://github.com/locuslab/robust_union/tree/master/CIFAR10666https://github.com/deepmind/deepmind-
research/tree/master/adversarial_robustness, and an adversarially-trained
model with respect to the LPIPS distance777See Section 4.2 for details. on
ImageNet [11]888https://github.com/cassidylaidlaw/perceptual-advex, and
compare the attack performance of PWCF and APGD999We implement the margin loss
on top of AutoAttack. [29] from the AutoAttack package. The attack radii
$\varepsilon$’s are set following the common practice of adversarial
RE101010E.g., https://robustbench.github.io/ for Cifar10 $\ell_{2}$ and
$\ell_{\infty}$; https://github.com/locuslab/robust_union for Cifar10
$\ell_{1}$; [11] for ImageNet $\ell_{2}$ and $\ell_{\infty}$..

From Table 1, we can conclude that: (1) PWCF performs strongly and comparably
to APGD on $\ell_{1}$, $\ell_{2}$ and $\ell_{\infty}$ attacks, especially
using _margin loss_ as the objective; (2) PWCF is weak on $\ell_{1}$ and
$\ell_{\infty}$ attacks using CE loss, likely due to the bad numerical scaling
of the CE loss; (3) Combining all successful attack samples found by APGD and
PWCF (APGD+PWCF) can further reduce the robust accuracy compared to any single
APGD or PWCF attack—PWCF and APGD are complementary. Note that [20] also
remarks that the diversity of solutions matters much more than the superiority
of individual solvers, which is the reason why AutoAttack includes Square
Attack–a zero-th order black-box attack method that does not perform strongly
itself as shown in Table 1.

###  4.2 PWCF works for general (almost everywhere) differentiable $\ell_{p}$
and non-$\ell_{p}$ metrics

As highlighted in Section 2, a major limitation of PGD-based methods is that
they cannot handle distances other than $\ell_{1}$, $\ell_{2}$, and
$\ell_{\infty}$. By contrast, PWCF stands out as a convenient choice for
general distances. To show this, we apply PWCF to solve Eq. 1 with
$\ell_{1.5}$ and $\ell_{8}$ distances. In addition, we also solve Eq. 1 with
the LPIPS perceptual metric [11, 30], i.e., perceptual attack (PAT) with

$\displaystyle
d(\mathbf{x},\mathbf{x}^{\prime})\doteq||\phi(\mathbf{x})-\phi(\mathbf{x}^{\prime})||_{2}~{}\text{,}\quad\phi(\mathbf{x})\doteq[~{}\widehat{g}_{1}(\mathbf{x}),\dots,\widehat{g}_{L}(\mathbf{x})~{}]$

where $\widehat{g}_{1}(\mathbf{x}),\dots,\widehat{g}_{L}(\mathbf{x})$ are the
vectorized intermediate feature maps from pretrained DNNs.

Table 2: Attack performance of PWCF with margin loss on general $\ell_{p}$ and non-$\ell_{p}$ metrics. We report attack success rates (numbers are in $\%$). We test on $\ell_{1.5}$, $\ell_{8}$, and PAT; numbers on $\ell_{1}$, $\ell_{2}$, and $\ell_{\infty}$ are included for reference. Numbers below each rate in parenthesis are the perturbation radii. | Special $\ell_{p}$ | General $\ell_{p}$
---|---|---
Model | $\ell_{1}$ | $\ell_{2}$ | $\ell_{\infty}$ | $\ell_{1.5}$ | $\ell_{8}$ | PAT
Clean | $100$ | $100$ | $100$ | $100$ | $100$ | $100$
| $(2400)$ | $(6.09)$ | $(0.01569)$ | $(44.40)$ | $(0.07)$ | $(0.5)$
PAT | $49.7$ | $40.7$ | $35.2$ | $100$ | $100$ | $100$
| $(2400)$ | $(4.7)$ | $(0.017)$ | $(443.98)$ | $(0.70)$ | $(0.5)$

PWCF handles them seamlessly, as shown in Table 2. Here we do not strive to
set the most reasonable perturbation radii, especially for $\ell_{1.5}$ and
$\ell_{8}$ that have not been tested before, and hence we also do not stress
the attack rates. Our point is that _PWCF is able to handle these general
$\ell_{p}$ distances_. Table 3 further summarizes the details of performing
the perceptual attack with $\varepsilon=0.5$. Existing methods to compare are
Perceptual Projected Gradient Descent (PPGD), Lagrangian perceptual attack
(LPA) and its variant fast Lagrangian perceptual attack (Fast-LPA) methods,
all developed in [11], based on iterative linearization and projection (PPGD),
or penalty method (LPA, Fast-LPA) respectively. In addition to the objective
values and attack success rates, we also report their chances of finding
infeasible solutions. As observed in Table 3, our PWCF is the clear winner.

Table 3: Performance comparison of different methods for solving PAT with the clipped CE and margin (M) loss. Viol. reports the ratio of final solutions that violate constraints. Succ. is the ratio of all _feasible successful attacks_ divided by _total number of samples_. The model we test is pat_alexnet_0.5 [11]. Evaluation is performed on ImageNet-100 dataset. | CE Objective | Margin Objective
---|---|---
Method | Viol. ($\%$) $\downarrow$ | Succ. ($\%$) $\uparrow$ | Viol. ($\%$) $\downarrow$ | Succ. ($\%$) $\uparrow$
Fast-LPA | $73.8$ | $3.54$ | $41.6$ | $56.8$
LPA | 0.00 | $80.5$ | 0.00 | $97.0$
PPGD | $5.44$ | $25.5$ | 0.00 | $38.5$
PWCF | $0.62$ | $93.6$ | 0.00 | $100$

##  5 Conclusion

In this paper, we propose PWCF to solve the maximization problem Eq. 1 in
robustness evaluations, blending the SOTA constrained optimization solver
PyGRANSO with constraint folding and other tweaks. Our experimental results
show that 1) PWCF can provide competitive and complementary performance
compared with the SOTA methods on $\ell_{1}$, $\ell_{2}$, and $\ell_{\infty}$
attacks; 2) PWCF can deal with general attack models such as $\ell_{p}$ with
$p>1$ and perceptual attacks, which are beyond the reach of PGD-based methods;
3) PWCF involves little to zero parameter-tuning and obtains reliable
solutions based on a principled stopping criterion. Our preliminary
experiments also show that the proposed PWCF is general enough to solve Eq. 2
with good quality, which we will present in forthcoming papers.

## References

  * [1] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus,  “Intriguing properties of neural networks,”  arXiv preprint arXiv:1312.6199, 2013.
  * [2] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy,  “Explaining and harnessing adversarial examples,”  in International Conference on Learning Representations, 2015.
  * [3] Dan Hendrycks and Thomas Dietterich,  “Benchmarking neural network robustness to common corruptions and perturbations,”  in International Conference on Learning Representations, 2018.
  * [4] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry,  “Exploring the landscape of spatial robustness,”  in Proceedings of the 36th International Conference on Machine Learning, Kamalika Chaudhuri and Ruslan Salakhutdinov, Eds. 09–15 Jun 2019, vol. 97 of Proceedings of Machine Learning Research, pp. 1802–1811, PMLR.
  * [5] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song,  “Spatially transformed adversarial examples,”  in International Conference on Learning Representations, 2018.
  * [6] Eric Wong, Frank R. Schmidt, and J. Zico Kolter,  “Wasserstein adversarial examples via projected sinkhorn iterations,”  arXiv:1902.07906, Feb. 2019.
  * [7] Cassidy Laidlaw and Soheil Feizi,  “Functional adversarial attacks,”  Advances in neural information processing systems, vol. 32, 2019\.
  * [8] Hossein Hosseini and Radha Poovendran,  “Semantic adversarial examples,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 1614–1619.
  * [9] Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, and David A Forsyth,  “Big but imperceptible adversarial perturbations via semantic manipulation,”  arXiv preprint arXiv:1904.06347, vol. 1, no. 3, 2019.
  * [10] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu,  “Towards deep learning models resistant to adversarial attacks,”  arXiv preprint arXiv:1706.06083, 2017.
  * [11] Cassidy Laidlaw, Sahil Singla, and Soheil Feizi,  “Perceptual adversarial robustness: Defense against unseen threat models,”  in ICLR, 2021.
  * [12] Francesco Croce and Matthias Hein,  “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,”  in International conference on machine learning. PMLR, 2020, pp. 2206–2216.
  * [13] Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin Vechev,  “Fast and effective robustness certification,”  Advances in neural information processing systems, vol. 31, 2018\.
  * [14] Francesco Croce and Matthias Hein,  “Minimally distorted adversarial examples with a fast adaptive boundary attack,”  in International Conference on Machine Learning. PMLR, 2020, pp. 2196–2205.
  * [15] Maura Pintor, Fabio Roli, Wieland Brendel, and Battista Biggio,  “Fast minimum-norm adversarial attacks through adaptive norm constraints,”  Advances in Neural Information Processing Systems, vol. 34, 2021\.
  * [16] Yihua Zhang, Guanhua Zhang, Prashant Khanduri, Mingyi Hong, Shiyu Chang, and Sijia Liu,  “Revisiting and advancing fast adversarial training through the lens of bi-level optimization,”  arXiv:2112.12376, Dec. 2021.
  * [17] Frank E Curtis, Tim Mitchell, and Michael L Overton,  “A bfgs-sqp method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles,”  Optimization Methods and Software, vol. 32, no. 1, pp. 148–181, 2017.
  * [18] Buyun Liang, Tim Mitchell, and Ju Sun,  “NCVX: A user-friendly and scalable package for nonconvex optimization in machine learning,”  arXiv:2111.13984, 2021.
  * [19] Marius Mosbach, Maksym Andriushchenko, Thomas Trost, Matthias Hein, and Dietrich Klakow,  “Logit pairing methods can fool gradient-based attacks,”  arXiv preprint arXiv:1810.12042, 2018.
  * [20] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin,  “On evaluating adversarial robustness,”  arXiv:1902.06705, Feb. 2019.
  * [21] Dimitri Bertsekas,  Nonlinear Programming 3rd Edition,  Athena Scientific, 2016.
  * [22] Gianni Pillo and Massimo Roma,  Large-scale nonlinear optimization, vol. 83,  Springer Science & Business Media, 2006.
  * [23] Andreas Wächter and Lorenz T Biegler,  “On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming,”  Mathematical programming, vol. 106, no. 1, pp. 25–57, 2006.
  * [24] Sören Laue, Matthias Mitterreiter, and Joachim Giesen,  “Geno–generic optimization for classical machine learning,”  Advances in Neural Information Processing Systems, vol. 32, 2019\.
  * [25] James V Burke, Frank E Curtis, Adrian S Lewis, Michael L Overton, and Lucas EA Simões,  “Gradient sampling methods for nonsmooth optimization,”  Numerical Nonsmooth Optimization, pp. 201–225, 2020.
  * [26] JRRA Martins and Nicholas MK Poon,  “On structural optimization using constraint aggregation,”  in VI World Congress on Structural and Multidisciplinary Optimization WCSMO6, Rio de Janeiro, Brasil. Citeseer, 2005.
  * [27] Pratyush Maini, Eric Wong, and Zico Kolter,  “Adversarial robustness against the union of multiple perturbation models,”  in International Conference on Machine Learning. PMLR, 2020, pp. 6640–6650.
  * [28] Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli,  “Uncovering the limits of adversarial training against norm-bounded adversarial examples,”  arXiv preprint arXiv:2010.03593, 2020.
  * [29] Francesco Croce and Matthias Hein,  “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,”  ArXiv, vol. abs/2003.01690, 2020.
  * [30] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang,  “The unreasonable effectiveness of deep features as a perceptual metric,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 586–595.