# Wavelet Regularization Benefits Adversarial Training

Jun Yan, Huilin Yin🖂, Xiaoyang Deng, Ziming Zhao, Wancheng Ge, Hao Zhang, and
Gerhard Rigoll Jun Yan, Huilin Yin (corresponding author), Xiaoyang Deng,
Ziming Zhao, Wancheng Ge, and Hao Zhang is with the College of Electronic and
Information Engineering at Tongji University, Shanghai, China (E-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>zhang_hao@tongji.edu.cn). Gerhard
Rigoll<EMAIL_ADDRESS>is with the Institute for Human-Machine Communication
in Technical University of Munich.

###### Abstract

Adversarial training methods are state-of-the-art (SOTA) empirical defense
methods against adversarial examples. Many regularization methods have been
proven to be effective with the combination of adversarial training.
Nevertheless, such regularization methods are implemented in the time domain.
Since adversarial vulnerability can be regarded as a high-frequency
phenomenon, it is essential to regulate the adversarially-trained neural
network models in the frequency domain. Faced with these challenges, we make a
theoretical analysis on the regularization property of wavelets which can
enhance adversarial training. We propose a wavelet regularization method based
on the Haar wavelet decomposition which is named Wavelet Average Pooling. This
wavelet regularization module is integrated into the wide residual neural
network so that a new WideWaveletResNet model is formed. On the datasets of
CIFAR-10 and CIFAR-100, our proposed Adversarial Wavelet Training method
realizes considerable robustness under different types of attacks. It verifies
the assumption that our wavelet regularization method can enhance adversarial
robustness especially in the deep wide neural networks. The visualization
experiments of the Frequency Principle (F-Principle) and interpretability are
implemented to show the effectiveness of our method. A detailed comparison
based on different wavelet base functions is presented. The code is available
at the repository: https://github.com/momo1986/AdversarialWaveletTraining.

###### Index Terms:

deep learning, robustness, adversarial training, wavelet transform, Lipschitz
constraint.

##  I Introduction

The existence of adversarial examples is the obstruction of the realization of
trustworthy deep learning. The neural networks with the high recognition
performance on the clean testing datasets would be vulnerable to the white-box
adversarial examples [1, 2, 3, 4, 5] which the adversaries have the knowledge
of the target models and the training data distribution and black-box
adversarial examples [6, 7, 8, 9, 10] which the adversaries have no access to
the internal structures of the neural networks.

Many empirical defense methods have been proposed to mitigate the adversarial
effects including distillation [11], randomization [12], guided denoising
[13], pixel defending [14], pixel masking [15], Bayesian deep learning [16],
and so on. Among these methods, adversarial training can be regarded as the
SOTA defense method to promote robustness in the optimization process of
neural networks [3, 17, 18, 19, 20, 21]. The core idea is to improve the
robust generalization ability in the min-max optimization process, in which
the perturbations should be as large as possible while the expected loss of
the model should be as small as possible. The “gradient penalty” effect [22]
can be achieved via the Nash equilibrium so that the neural network models are
satisfied with the small Lipschitz constraint. The proposal and development of
adversarial training methods help improve the performance of the neural
network model under adversarial attacks.

To realize the minimization of adversarial empirical risk, the regularization
tricks would be combined with adversarial training methods to boost
robustness, including logit pairing [23], parametric noise injection [24],
early stopping [18, 20], weight decay [20], Jacobian regularization [25], and
the utilization of CutMix method [26]. All these methods are implemented in
the time domain, which may ignore the phenomenon that the adversarial example
is a high-frequency phenomenon [27, 28]. Therefore, it is worthwhile to
regulate adversarial training in the frequency domain to boost adversarial
robustness [29, 30, 27, 31]. The wavelet has properties of smoothness,
approximation, and reproduction of polynomials [32, 33] which is suitable to
handle nonstationary signals. It is a potential regularization method to
handle high-frequency and nonstationary adversarial perturbations.

Figure 1: Our Adversarial Wavelet Training framework. Fig. (a) is the data
pipeline with the adversary generation on the clean dataset. Fig. (b)
illustrates the Adversarial Wavelet Training method which is incorporated
adversarial training and wavelet regularization can boost the robustness of
DNNs (Fig.(b2) is cited from the blog article [34] @Towards Data Science).
Fig. (c) demonstrates that our proposed method can realize the robustness
guarantee. Moreover, the learning feature of our proposed method is
interpretable with the visualization of the class activation mapping (CAM)
method. From the perspective of F-Principle, our proposed method realizes a
low high-frequency error in the Fourier heat map.

Since the high-frequency perturbation may lead to the adversarial
vulnerabilities [27, 28], the utilization of the wavelet would be a feasible
regularization method. In this paper, we propose a framework combined with
wavelet transform and adversarial training. As Fig.1 illustrates, the
Adversarial Wavelet Training method incorporates an adversarial training
framework with the regularization method of wavelet function implemented in
the frequency domain. For our proposed method, we evaluate the clean accuracy
and the robust accuracy on the CIFAR-10 and CIFAR-100 datasets [35]. We also
give an elaborate theoretical analysis of our adversarial wavelet training
method that such a framework satisfies the Lipschitz constraint.

Our contributions of this paper are listed as follows:

  * •

We make the theoretical analyses about the robustness under the framework
combined with the wavelet analysis and adversarial training for the proposal
of adversarial wavelet training.

  * •

Our proposed wavelet regularization method based on adversarial training
realizes the considerable robustness under the different types of attacks
including white-box attacks, black-box attacks, adaptive attacks, and auto
attacks on the CIFAR-10 dataset and CIFAR-100 dataset [35].

  * •

We make an empirical comparison between different wavelet base functions to
prove that the Haar base function is the most suitable.

This paper is organized as follows. The related work is introduced in Section
II. In Section III, we make a theoretical analysis to show the effectiveness
of wavelet transform as a regularization method for adversarial training. Our
proposed method is illustrated in Section IV. The experiment detail and result
would be described in Section V. Finally, Section VI concludes our research.

##  II Related Work

In this section, we would have a literature review on the work related to
adversarial robustness. The adversarial defense methods against the
adversarial attacks would be introduced. Moreover, we would focus on the
utilization of wavelet methods in deep learning.

###  II-A Adversarial Robustness: Attack And Defense

In the field of computer vision, the adversarial examples that the original
clean images are combined with the imperceptible noises can fool the neural
visual systems. In the category of white-box attacks, the adversaries could
query the information of model structures and model parameters. The search for
minimal adversarial perturbations can be modeled as an optimization problem
[1, 5], a gradient perturbation problem [2, 4, 3], or a classification
hyperplane problem [36]. However, in the real applications of artificial
intelligence (AI), the knowledge of the models cannot be obtained. The
research on the black-box attacks with the manipulation of the model
input/output (I/O) is more universal and realistic in such a scenario. A lot
of methods have been proposed on such a research topic including the attacks
based on the transferability [37, 38], gradient estimation [6, 10, 7],
decision [39, 40], and sampling [9]. On the evaluation of adversarial
robustness, only the empirical study on the static attack methods [2, 3] is
insufficient. Then, the adaptive attack methods [5, 41, 42] are proposed to
break the defense methods with counter-countermeasures. To fix the pitfalls of
improper tuning of hyperparameters of the attacks, gradient obfuscation, or
masking in the robustness evaluation, the auto-attack method [43] with the
extension of the multi-step iterative attack method [3] can provide a fair
evaluation of the adversarial robustness. The existence of adversarial
examples is a huge challenge but also a great promise for trustworthy deep
learning.

To handle the security threat of deep learning, many empirical defense methods
have been proposed. The adversarial attack detection methods can reject the
adversarial examples to avoid deceit. The detection methods can be built on
the support vector machines (SVMs) [44], SVMs with Discrete Wavelet Transform
and Discrete Sine Transform [45], deep Bayesian classifier [46], variational
autoencoder (VAE) [47], perturbation rectifying network [48], local intrinsic
dimensionality [49], K-density detector with Reverse Cross-Entropy [50],
Fisher information matrix [51], nonlinear autoencoder (AE) [52], and so on.
The attack mitigation methods are inclined to the inference on the adversarial
examples rather than the direct rejection. The input-transformation methods
[53, 54, 55] can handle the black-box noise attacks but shrivel under the
white-box adversarial attacks [41]. The recent work of pixel masking [15]
explores cognitive data augmentation to improve robustness under white-box
adversarial attacks. The ensemble methods [56, 57], the randomization methods
[12, 16], and the combination of the ensemble method and the randomization
method [58] are proposed to realize the security by obscurity. The
distillation method [11] and the pruning method [59] attempt to realize the
adversarial robustness through the model compression.

Among the empirical defense methods, adversarial training [3, 17, 60, 61, 62,
63, 20, 19, 21] is considered the feasible scheme against adversarial attacks.
The restraint on the gradients is validated to be beneficial to the
adversarial robustness [3]. Many pieces of research related to adversarial
training have been delivered including the realization of a better trade-off
between clean accuracy and robust accuracy [17, 64, 65, 66], its combination
with semi-supervised learning [61], curriculum learning [19], or ensemble
method [60], and reduction on the training cost [63, 62].

At the model level, most adversarial training methods are implemented on the
WideResNet model [67]. Diverse pieces of theoretical literature show that the
wide model can find the approximately-global-optimal minima during the
optimization process [68, 69, 70] with the learning dynamics of a linear model
[71, 72]. The overparameterization phenomenon of WideResNet would make the
loss surface of deep learning more smooth and could transform the non-convex
optimization problem into the “analogous” convex problem. The theoretical
foundation can be referred to the work on learning theory [73, 74, 75, 76].
The recent research via the tool of tensor interpolation [77] and isoperimetry
[78] also supports the conjecture that the overparameterization is the
robustness guarantee of many deep learning models including WideResNet. It is
counterintuitive in the context of structural risk minimization (SRM).
Moreover, the “overfitting” problem is a spiny problem for the realization of
adversarial empirical risk minimization (ERM) during the defense process.
Thus, it incurs a need to utilize regularization techniques in adversarial
training to restrain the solution space. The recent research from the
perspective of neural tangent kernel (NTK) [79] suggests that the
regularization method is still essential to tune the perturbation stability of
deep wide neural networks under the attacks despite the intrinsic robust
property of the deep wide neural model structures. It breeds the necessity of
the research of regularization methods in the adversarial training of wide
deep neural networks.

Currently, early stopping and weight decay are two effective regularization
methods in adversarial training [18, 20]. Other regularization methods like
logit pairing [23], parametric noise injection [24], curvature regularization
[80], and Jacobian regularization [25] can be combined with adversarial
training. Not long ago, it has been proven that the CutMix method [81] can
also help boost adversarial robustness [26]. However, all these methods are
implemented in the time domain. The Fourier perspective [27, 28] provides an
illustration of the adversarial vulnerability. Therefore, it inspires us to
provide a regularization method for adversarial training in the frequency
domain.

###  II-B Wavelet On Deep Learning

Wavelet analysis is an important research field of modern mathematics [33, 82,
32] that inspires wireless communication, image science, and quantum physics.
In the field of statistical learning, researchers were devoted to analyzing
the multi-scale property of the wavelet on trees, graphs, and other high-
dimensional data for a further combination of semi-supervised learning or
supervised learning [83, 84]. The wavelet transform is a feasible method to
achieve the regularization [33, 32].

After the renaissance of deep learning, some research achievements are at the
intersection of wavelet analysis and deep learning, including the method of
group invariant scattering based on non-orthogonal Morlet wavelet [85, 86] or
orthogonal Haar wavelet [87], the CNN models based on the multi-resolution
character of wavelet [88, 89], wavelet pooling [90], and deep learning
denoising method based on the wavelet [91]. The robustness of wavelet against
noise [91] and its down-sampling property in the frequency domain [90] inspire
us to explore the application of wavelet in the research of adversarial
training.

However, all these methods are used in the field of natural training and never
extended to robust learning. The work of wave transform attack [92] is an
exception that combines the wavelet method with the research of adversarial
examples. Such a study demonstrates the feasibility of adversarial robustness
on wavelet. Therefore, we are devoted to combining the adversarial training
with the wavelet method to utilize the character of wavelet: multi-resolution
analysis, and regularization. Besides these motivations, wavelet transform has
also connections with the Frequency Principle (F-Principle) [27] in deep
learning. The traditional Fourier transform is coarse-grained and global. In
contrast, the multi-resolution property of the wavelet transform can aggregate
the low-frequency elements and the high-frequency elements in a more rational
way to promote robustness under the perturbations.

##  III Theoretical Preliminary

###  III-A Wavelet Transform As Regularization

Our analysis is inspired by the pure mathematics research [33, 32], and we
extended it to adversarial deep learning. The robustness of neural networks
can be described in the concept of “Lipschitz constraint”.

######  Definition III.1.

(Lipschitz continuity) If $\left|x_{1}-x_{2}\right|\leqslant\varepsilon$, and
$\left|f\left(x_{1}\right)-f\left(x_{2}\right)\right|\leqslant
L\left|x_{1}-x_{2}\right|$, then the function $f(x)$ satisfies with the
Lipschitz constant $L$, $0<L<\infty$.

The robust optimization of neural network is based on a locally considerable
Lipschitz constraint ($L$ should be as small as possible). Such a constraint
could keep the stability of the system under perturbation.

######  Definition III.2.

(Continuous wavelet transform) There exist a dilation factor “$a$” and a
translation factor “$b$” which vary continuously over the real number space
$\mathbb{R}$, the continuous wavelet transform is given by “resolution
identity” formula defined in Eq. (1):

$f=C_{\psi}^{-1}\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}\frac{dadb}{a^{2}}\left\langle
f,\psi^{a,b}\right\rangle\psi^{a,b},$ (1)

where the wavelet base is
$\psi^{a,b}(x)=|a|^{-1/2}\psi\left(\frac{x-b}{a}\right)$, and $(,)$ denotes
the $L^{2}$-inner product defined in the Hilbert space. For the constant
$C_{\psi}$, it is determined by the base $\psi$ and defined in Eq. (2):

$C_{\psi}=2\pi\int_{-\infty}^{\infty}d\xi|\hat{\psi}(\xi)|^{2}|\xi|^{-1}.$ (2)

A doubly-indexed family of wavelets from $\psi$ by dilating and translating is
generated by Eq. (3):

$\psi^{a,b}(x)=|a|^{-1/2}\psi\left(\frac{x-b}{a}\right),$ (3)

where $a,b\in\mathbb{R},\ a\neq 0$. The normalization of base is
$\left\|\psi^{a,b}\right\|=\|\psi\|\ for\ a,\ b$ with the assumption of
$\|\psi\|=1$. The continuous wavelet form can be written as the form of Eq.
(4):

$\displaystyle\left(T^{\text{wav }}f\right)(a,b)$ $\displaystyle=\left\langle
f,\psi^{a,b}\right\rangle$ (4) $\displaystyle=\int
dxf(x)|a|^{-1/2}\psi\left(\frac{x-b}{a}\right).$

Note that $\left(T^{\operatorname{wav}}f\right)(a,b)\mid\leq\|f\|$.

The study of computer vision is based on discrete space. The introduction of
the continuous wavelet transform could be a preliminary for the further
description of the discrete wavelet transform.

The theorems below were previously depicted in the book of Daubechies [33],
some theorems are ameliorated to describe the scenario of adversarial
robustness clearly while others are directly derived from the research
achievement of pure mathematics.

######  Theorem III.1.

(Restated Lipschitz constraint of wavelet base in the Hölder space): There
exists an input feature variable $x$ with the perturbation $\delta$, suppose
that $\int dx(1+|x|)|\psi(x)|<\infty$ and the Fourier transform
$\hat{\psi}(x)$ has the property $\hat{\psi}(0)=0$. The bounded function $f$
in the Hölder space is continuous with the exponent $\alpha$ ($0<\alpha\leq
1$) as defined in Eq. (5):

$|f(x+\delta)-f(x)|\leq C|\delta|^{\alpha}.$ (5)

Then, its wavelet transform is satisfied with Eq. (6):

$\left|T^{\text{wav }}(a,b)\right|=\left|\left\langle
f,\psi^{a,b}\right\rangle\right|\leq C^{\prime}|a|^{\alpha+1/2}.$ (6)

The Theorem III.1 derives a theoretical framework that wavelet base is
Lipschitz-continuous in the Hölder space. It supports the property of the
local stability of the wavelet base. The related proof would be provided in
the appendix.

######  Theorem III.2.

(Compact support of wavelet base) [33] Suppose that the wavelet base $\psi$ is
compactly supported, the universal function $f\in L^{2}(\mathbb{R})$ is
bounded and continuous with the exponent $\alpha,\ 0<\alpha\leq 1$. Then, the
wavelet transform of $f$ is satisfied with Eq. (7):

$\left|\left\langle f,\psi^{a,b}\right\rangle\right|\leq C|a|^{\alpha+1/2},$
(7)

where $f$ is Hölder continuous with the exponent $\alpha$.

The Theorem III.2 provides an upper bound of the transform of a compactly
supported wavelet base. Generally speaking, such an upper bound determines the
mutation point of the signal. The proof would be provided in the appendix.

Both the Theorem III.1 and the Theorem III.2 indicate that the Hölder
continuity of a function can be characterized by the decay value $\alpha$ of
the wavelet transform. Moreover, the wavelet transform can also be used to
characterize the local regularity of the signal as a mathematical zoom, which
surpasses the windowed Fourier transform. It helps wavelet transform avoid the
aliasing effect of high-frequency signals and singular signals.

######  Theorem III.3.

(Restated local regularity of wavelet transform) Suppose that $\int
dx(1+|x|)|\psi(x)|<\infty$ and $\int dx\psi(x)=0$. If a bounded function $f$
is Hölder continuous in $x_{0}$, with the exponent $\alpha\in[0,1]$. On the
robustness problem with the perturbation $\delta$, there exists an inequation
$\left|f\left(x_{0}+\delta\right)-f\left(x_{0}\right)\right|\leq
C|\delta|^{\alpha}$. Furthermore, it can be deduced that $\left|\left\langle
f,\psi^{a,x_{0}+b}\right\rangle\right|\leq
C|a|^{1/2}\left(|a|^{\alpha}+|b|^{\alpha}\right)$.

The Theorem III.3 illustrates the local regularity of the wavelet transform.
It is a further derivation of Theorem III.1 with a different restricted
condition which supports the property of local regularity of wavelet base. The
corresponding proof would be provided in the appendix.

######  Theorem III.4.

(Upper bound of wavelet transform inside the Hölder space) [33] Suppose that
the wavelet base $\psi$ is compactly supported, the function $f\in
L^{2}(\mathbb{R})$ is bounded and continuous with the variable of $\gamma>0$
and $\alpha\in[0,1]$, there exists a formula defined in Eq. (8):

$\begin{array}[]{c}\left|\left\langle f,\psi^{a,b}\right\rangle\right|\leq
C|a|^{\gamma+1/2}\quad\text{ uniformly in }b,\\\ \left|\left\langle
f,\psi^{a,b+x_{0}}\right\rangle\right|\leq
C|a|^{1/2}\left(|a|^{\alpha}+\frac{|b|^{\alpha}}{|\log|b||}\right),\end{array}$
(8)

where $f$ is Hölder continuous with the exponent $\alpha$.

The Theorem III.4 derives an upper bound of the wavelet transform inside the
Hölder space. Both Theorem III.2 and Theorem III.4 show that compactly-
supported wavelet bases have strong regularization constraints. The related
proof would be provided in the appendix.

######  Theorem III.5.

(Restated Lipschitz constraint of wavelet in the multi-resolution space)
Suppose that the perturbation is $2^{-(j+1)}\leq\delta\leq 2^{-j}$, there
exists a variable $l\in N$ which satisfies one of the two followings
$(\ell-1)2^{-j}\leq x\leq x+\delta\leq\ell 2^{-j}$ or $(\ell-1)2^{-j}\leq
x\leq\ell 2^{-j}\leq x+\delta\leq(\ell+1)2^{-j}$. Further, it can leads to the
Hölder continuity with the exponent $\alpha$ defined in Eq. (9):

$|f(x)-f(x+\delta)|\leq C^{\prime}2^{-\alpha j}\leq
C^{\prime\prime}|\delta|^{\alpha}.$ (9)

The Theorem III.5 gives an illustration of the Lipschitz continuity in the
“multi-resolution” space. The proof would be provided in the appendix. Such a
theorem can be the theoretical support of the “mathematical microscope”
character of the wavelet. This property is beneficial to extract the detailed
information of object during the recognition. Also, the regularity properties
of wavelet base functions based on the Lipschitz constraint would assist the
adversarial training method to boost the robustness of models. The related
proof would be provided in the appendix.

###  III-B Multi-resolution Analysis Of Wavelet

For the wavelet transform function $f$, there exist different resolutions. We
know that the resolution $2^{j-1}$ is more fine-grained than the resolution
$2^{j}$. The approximation with different resolutions can be the linear
combination of the wavelet base $\psi_{j,k}$. A ladder of the space
$\left(V_{j}\right)_{j\in\mathbb{Z}}$ can represent the successive resolution
levels, where $V_{j}=\left\\{f\in L^{2}(\mathbb{R})\right\\}$ and $f$ is
piecewise constant on the $\left[2^{\jmath}k,2^{\jmath}(k+1)\right]$. These
spaces have following properties:

1) Monotonicity: $\cdots\subset V_{2}\subset V_{1}\subset V_{0}\subset
V_{-1}\subset V_{-2}\subset\cdots.$

2) Uniqueness: $\bigcap_{j\in\mathbb{Z}}V_{j}=\\{0\\}$.

3) Density: $\overline{\bigcup_{j\in\mathbb{Z}}V_{j}}=L^{2}(\mathbb{R})$.

4) Multi-scale: $f\in V_{j}\leftrightarrow f\left(2^{j}.\right)\in V_{0}$.

5) Symmetry: $f\in V_{0}\rightarrow f(\cdot-n)\in V_{0}$ for all
$n\in\mathbb{Z}$.

With the “multi-resolution” character, the wavelet transform could do the
projection of the signal to space with different wavelet base scales.
Therefore, the signal could be decomposed in different resolutions. In the
task of image classification, the coarse shape can be represented by the low
resolution with a large scale, and the texture detail can be represented by
the high resolution with a small scale. In the task of object detection [93],
the multi-scale technology may boost performance by generating feature maps
with different scales. For engineering practice, the multi-scale property is
beneficial for almost all the tasks of computer vision.

###  III-C Wavelet Transform On Images

Eq. (3) derives the representation formula of the continuous wavelet
transform. In reality, a variable $a$ would be discretized that $a=a_{0}^{m}$,
for $m\in\mathbb{Z}$ while the variable of $b$ can be described as
$b=nb_{0}a_{0}^{m}$. Then, a new discretized wavelet base can be formulated in
Eq. (10):

$\psi_{m,n}(x)=a_{0}^{-m/2}\psi\left(\frac{x-nb_{0}a_{0}^{m}}{a_{0}^{m}}\right)=a_{0}^{-m/2}\psi\left(a_{0}^{-m}x-nb_{0}\right).$
(10)

It leads to the discrete wavelet transform. The discrete wavelet coefficient
$\left\langle f,\psi_{m,n}\right\rangle$ can approximately characterize the
function $f$ or reconstruct $f$ in a numerically stable way from the
$\left\langle f,\psi_{m,n}\right\rangle$. $f$ can also be written as a
superposition of the combination $\psi_{m,n}$. In the framework of discrete
wavelet transform, the reconstruction problem is defined in Eq. (11):

$f=\sum_{m,n}\left\langle f,\psi_{m,n}\right\rangle\widetilde{\psi_{m,n}}.$
(11)

For the wavelet base function $\psi$, there exists a scaling function
$\varphi$, the 2D representation formula can be defined in Eq. (12):

$\displaystyle\varphi_{j,m,n}(x,y)$
$\displaystyle=2^{j/2}\varphi\left(2^{j}x-m,2^{j}y-n\right),$ (12)
$\displaystyle\psi_{j,m,n}^{i}(x,y)$
$\displaystyle=2^{j/2}\psi^{i}\left(2^{j}x-m,2^{j}y-n\right),$

where index $i$ identifies the directional wavelets. The discrete wavelet
transform of image $f(x,\ y)$ of size $M\times N$ is then defined in Eq. (13):

$\begin{array}[]{l}W_{\varphi}\left(j_{0},m,n\right)=\frac{1}{\sqrt{MN}}\sum_{x=0}^{M-1}\sum_{y=0}^{N-1}f(x,y)\varphi_{j_{0},m,n}(x,y),\\\
W_{\psi}^{i}(j,m,n)=\frac{1}{\sqrt{MN}}\sum_{x=0}^{M-1}\sum_{y=0}^{N-1}f(x,y)\psi_{j,m,n}^{i}(x,y),\end{array}$
(13)

where $j_{0}$ is an arbitrary starting scale, the scale $j$ satisfies $j\geq
j_{0}$.

##  IV Proposed Method

###  IV-A Haar Wavelet Function

The Haar wavelet base function defined in Eq. (14) is found by the French
mathematician A. Haar [94]. Haar function is symmetrical, double-orthogonal,
and compactly supported with the filter length of $2$ and vanishing movement
of $1$. Utilizing the Haar function can help extract the pattern robustly
without redundancy due to its orthogonality property as some classical
research pointed out [95]. Its support size is $1$, which is beneficial for
the operation of sparse compression. Moreover, since the Haar wavelet is
symmetrical, it can be adapted to the reconstruction of the signal.

$\psi(x)=\left\\{\begin{array}[]{rl}1&0\leq x<\frac{1}{2},\\\
-1&\frac{1}{2}\leq x<1,\\\ 0&\text{ otherwise.}\end{array}\right.$ (14)

Its scaling function can be formulated as the form of Eq. (15):

$\phi(t)=\left\\{\begin{array}[]{ll}1&0\leq t<1,\\\ 0&\text{ otherwise.
}\end{array}\right.$ (15)

The filter function $h[n]$ is defined as the form of Eq. (16):

$h[n]=:\left\\{\begin{array}[]{ll}\frac{1}{\sqrt{2}}&\text{ if
}\mathrm{n}=0,1,\\\ 0&\text{ otherwise. }\end{array}\right.$ (16)

The Haar transform can be written as the form of Eq. (17):

$\displaystyle\frac{1}{\sqrt{2}}\psi\left(\left(\frac{t}{2}\right)\right)$
$\displaystyle=\sum_{n=-\infty}^{\infty}(-1)^{1-n}h[1-n]\phi(t-n)$ (17)
$\displaystyle=\frac{1}{\sqrt{2}}(\phi(t-1)-\phi(t)).$

In image science, the utilization of wavelets helps concentrate the signal
energy and control the noises of sub-bands. The property of the wavelet is
also consistent with the logarithmic characteristics of the human visual
system. Among various wavelet bases, the effectiveness of the simple Haar base
function has been validated in the computer vision task [95]. The default
wavelet regularization method is based on the Haar function, however, its
feasibility would be analyzed in a comparison experiment with diverse bases.

###  IV-B Aggregated Wavelet Decomposition

Two-dimensional discrete wavelet transform (DWT) can be realized with the
digital filter and down-sampling module in the reality. By performing a
second-order decomposition in the wavelet domain according to the fast wavelet
transform (FWT) [96], the DWT output of $j$-scale coefficient
$W_{\varphi}[j,n]$ is $W_{\varphi}[j+1,k]$ with $n=2k$ as illustrated in Eq.
(18). Fig. 2 describes the wavelet decomposition process.

$\begin{array}[]{l}W_{\varphi}[j+1,k]=\left.h_{\varphi}[-n]*W_{\varphi}[j,n]\right|_{n=2k,k\leq
0},\\\ W_{\psi}[j+1,k]=\left.h_{\psi}[-n]*W_{\psi}[j,n]\right|_{n=2k,k\leq
0}.\end{array}$ (18)

Figure 2: The schematic diagram of the wavelet decomposition.

Figure 3: The Haar decomposition on the image sampled from CIFAR-10 dataset.
Fig. (a) denotes the first original CIFAR-10 image. Fig. (b) denotes its
corresponding approximation, horizontal detail, vertical detail, and diagonal
detail.

On the F-Principle in deep learning [27], biasing the high-frequency elements
towards the low-frequency domain would be better than the filtering operation.
Therefore, our wavelet decomposition strategy avoids discarding high-frequency
elements. It is assumed that the generated matrix of DWT is $\mathbf{W_{LL}}$,
$\mathbf{W_{LH}}$, $\mathbf{W_{HL}}$, and $\mathbf{W_{HH}}$, it corresponds to
the approximation, horizontal detail, vertical detail, and diagonal detail of
image sampled from the CIFAR-10 dataset as Fig. 3 demonstrates. The
utilization of the multi-resolution property of the wavelet would be
intuitively better on a wider frequency band than on a narrower frequency
band. To follow the F-Principle and utilize the multi-resolution property
efficiently, averaging the elements of the wavelet transform is more
appropriate than filtering the high-frequency elements. Thus, the down-
sampling operation is defined as the form of Eq. (19). We named it Wavelet
Average Pooling:

$\mathbf{W_{WaveletAveragePooling}}=(\mathbf{W_{LL}+W_{LH}+W_{HL}+W_{HH}})\cdot
0.25.$ (19)

Figure 4: The structure of Wide Residual Neural Network on the Wavelet. Fig.
(a) illustrates the basic residual elements derived from ResNet [97] and
WideResNet [67]. Fig. (b) illustrates our proposed network structure.

The method proposed in the previous research [90] has been utilized to improve
the normal generalization ability without adversarial perturbation. The FWT is
utilized in the previous method for forward propagation, and the inverse FWT
is operated as a reverse process in backpropagation. As a comparison, our
Wavelet Average Pooling would only do the operation of down-sampling to
regularize the neural network without the process of backpropagation. It has
two advantages compared to the previous method [90]: 1) It is convenient in
the computation; 2) Averaging on the condensed feature map could realize the
trade-off between low-frequency elements related to the generalization without
the perturbations and high-frequency elements related to adversarial
perturbations.

###  IV-C Robust Deep Wide Neural Network On Wavelet

The overparameterization of WideResNet [67] is beneficial for robustness when
the perturbation factor is considered in deep learning. It is analyzed in the
theoretical view [71, 69, 68, 72, 73, 74, 75, 76] and verified in a detailed
empirical work [20]. The overparameterized deep wide neural networks could
achieve a stable global minimum via the perspective of dynamical stability
[70], NTK [75], and modeling of the Gaussian random initialization [76]. The
generalization error of the wide residual neural network for the minimum-norm
solution is comparable to the Monte Carlo rate due to the phenomenon of
overparameterization [98]. If the square root of the parameter number of a
shallow wide neural network exceeds the size of training data, the stochastic
gradient descent with the random initialization can converge at a geometric
rate to a nearby global optima [99]. Moreover, via the tool of isoperimetry,
the randomly-distributed points on the surface of a sphere need a smooth curve
with the additional parameters for the fitting [78]. It derives a universal
law of robustness that good memorization needs overparameterized networks with
a large width and depth. All of the theoretical findings support the
motivation to explore the usage of wavelet regularization on the WideResNet
model.

Although deep wide neural networks are beneficial to adversarial robustness,
the explicit regularization methods are still required to achieve suitable
perturbation stability [79]. Several regularization tricks in the time domain,
e.g., early stopping and weight decay, were applied to the adversarial
training and testified to be effective [20]. Our wavelet regularization method
is implemented in the frequency domain.

Based on these motivations, we propose a wide deep residual neural network on
the wavelet. We named it WideWaveletResNet. As Fig. 4 illustrates, after the
propagation of the robustness against adversarial perturbation with three
residual blocks and the ReLU module, our proposed wavelet decomposition module
regularizes to do a down-sampling operation on the signal and satisfy the
small Lipschitz constraint. After the operation of average pooling (Average
Pooling) with the kernel size of 4, fully connected layer (FCN), and softmax
layer, the inference prediction with confidence can be exported. The
WideWaveletResNet with the module of wavelet regularization can promote
robustness with the adversarial training method. The experiment result would
validate the effectiveness of the wavelet regularization method. The lifting
is more intuitive in a wide and deep neural network.

##  V Experiment And Results

###  V-A Implementation Details

We train our WideWaveletResNet model and test our proposed method based on the
datasets of CIFAR-10 and CIFAR-100 [35]. The adversarial perturbation is
generated via the projected gradient descent (PGD) [3] algorithm with a step
size of 10. The learning rate schedule is in multi-steps with an initial value
of 0.1. The batch size is 128. The attack is in the $l_{\infty}$ norm
constraint. The perturbation budget $\varepsilon$ is $0.031$ (approximately
equal to $8/255$). The WideWaveletResNet model would be fit by the stochastic
gradient descent with the momentum and weight decay of $0.0005$. The trick of
“Early Stopping” mechanism analyzed by the previous work [18] is utilized. We
follow some useful tricks analyzed in the previous research [20]. The training
time of our Adversarial Wavelet Training method would take around 55h for 110
epochs on a server with a single NVIDIA-RTX 3090 GPU.

Figure 5: The loss curve and gradient curve of our adversarial training method
on the CIFAR-10 dataset. Fig. (a) illustrates the loss curve. Fig. (b)
illustrates the gradient curve.

Figure 6: The loss curve and gradient curve of our adversarial training method
on the CIFAR-100 dataset. Fig. (a) illustrates the loss curve. Fig. (b)
illustrates the gradient curve.

Fig. 5-(a) and Fig. 6-(a) give the demonstrations of loss curves for our
Adversarial Wavelet Training method on the CIFAR-10 and CIFAR-100 datasets.
With the wavelet regularization on the wide residual neural network during the
adversarial training process, the loss surfaces of our proposed method are
smooth. Empirically, the “over-fitting” phenomenon can be avoided if the
training loss curve is smooth enough. As Fig. 5-(b) and Fig. 6-(b) illustrate,
the gradients of our proposed method are not too huge on both CIFAR-10 dataset
and CIFAR-100 dataset. The adversarial wavelet training method boosts the
robustness via the mechanism of the “gradient penalty”. Our wavelet
regularization method restrains the gradient to realize a small Lipschitz
constraint ($L$ is not large) on the basis of adversarial training. The
experiment result further verifies that the wavelet regularization enhances
the adversarial training method.

###  V-B Experiment Of Generalization And Robustness

TABLE I: THE RECOGNITION PERFORMANCE (%) OF DIFFERENT DEFENSE METHODS AGAINST WHITE-BOX ATTACKS ON CIFAR-10 ($\varepsilon=0.031$). | Adversarial Training? | Clen Acc | FGSM Acc | PGD Acc | MIM Acc | C&W Acc |
---|---|---|---|---|---|---|---
Vanilla WideResNet-34-10 [67] | No |  90.04 $\pm$ 1.042 |  7.57 $\pm$ 2.209 |  0.00 $\pm$ 0.000 |  0.00 $\pm$ 0.004 |  0.00 $\pm$ 0.000 |
Wavelet CNN [88] | No |  89.09 $\pm$ 0.046 |  44.50 $\pm$ 0.240 |  24.35 $\pm$ 0.240 |  24.80 $\pm$ 0.334 |  24.55 $\pm$ 0.337 |
Scattering Network [86] | No |  91.33 $\pm$ 0.138 |  45.17 $\pm$ 0.937 |  23.09 $\pm$ 0.568 |  23.81 $\pm$ 0.600 |  23.36 $\pm$ 0.619 |
Deep Adaptive Wavelet Network [89] | No |  93.08 $\pm$ 0.167 |  42.27 $\pm$ 0.982 |  9.94 $\pm$ 0.712 |  10.78 $\pm$ 0.528 |  10.35 $\pm$ 0.548 |
Classical Adversarial Training [3] | Yes |  78.24 $\pm$ 0.331 |  52.84 $\pm$ 0.442 |  50.14 $\pm$ 0.293 |  49.22 $\pm$ 0.440 |  47.89 $\pm$ 0.189 |
Fast Adversarial Training [62] | Yes |  81.37 $\pm$ 0.207 |  53.09 $\pm$ 0.199 |  47.12 $\pm$ 0.131 |  47.45 $\pm$ 0.149 |  46.13 $\pm$ 0.123 |
Free Adversarial Training [63] | Yes |  82.61 $\pm$ 0.266 |  54.97 $\pm$ 0.258 |  49.60 $\pm$ 0.184 |  49.65 $\pm$ 0.151 |  48.65 $\pm$ 0.169 | .
TRADES [17] | Yes |  75.38 $\pm$ 1.219 |  50.34 $\pm$ 1.100 |  46.90 $\pm$ 0.790 |  47.39 $\pm$ 0.813 |  44.27 $\pm$ 0.836 |
Friendly Adversarial Training [19] | Yes |  86.47 $\pm$ 0.252 |  60.71 $\pm$ 0.150 |  51.56 $\pm$ 0.458 |  53.24 $\pm$ 0.404 |  51.84 $\pm$ 0.285 |
Adversarial Training Trick [20] | Yes |  87.25 $\pm$ 0.334 |  61.45 $\pm$ 0.339 |  54.45 $\pm$ 0.650 |  54.30 $\pm$ 0.451 |  53.88 $\pm$ 0.218 |
Adt-Exp [21] | Yes |  87.35 $\pm$ 0.117 |  60.14 $\pm$ 0.531 |  50.98 $\pm$ 0.354 |  52.35 $\pm$ 0.328 |  52.01 $\pm$ 0.122 |
Adt-Expam [21] | Yes |  87.66 $\pm$ 0.110 |  63.78 $\pm$ 0.349 |  52.64 $\pm$ 0.240 |  53.75 $\pm$ 0.251 |  53.16 $\pm$ 0.232 |
Adt-Impam [21] | Yes |  88.14 $\pm$ 0.139 |  67.06 $\pm$ 0.823 |  52.10 $\pm$ 0.108 |  52.64 $\pm$ 0.253 |  52.61 $\pm$ 0.097 |
Adversarial Wavelet Training (ours) | Yes |  87.34 $\pm$ 0.085 |  61.61 $\pm$ 0.361 |  54.76 $\pm$ 0.116 |  54.47 $\pm$ 0.602 |  54.25 $\pm$ 0.074 |
TABLE II: THE RECOGNITION PERFORMANCE (%) OF DIFFERENT DEFENSE METHODS AGAINST WHITE-BOX ATTACKS ON CIFAR-100 ($\varepsilon=0.031$). | Adversarial Training? | Clen Acc | FGSM Acc | PGD Acc | MIM Acc | C&W Acc
---|---|---|---|---|---|---
Vanilla WideResNet-34-10 [67] | No |  68.68 $\pm$ 1.279 |  4.11 $\pm$ 0.545 |  0.00 $\pm$ 0.000 |  0.00 $\pm$ 0.008 |  0.00 $\pm$ 0.000
Wavelet CNN [88] | No |  59.85 $\pm$ 0.237 |  18.43 $\pm$ 0.230 |  9.71 $\pm$ 0.163 |  9.92 $\pm$ 0.162 |  9.52 $\pm$ 0.183
Scattering Network [86] | No |  68.98 $\pm$ 0.204 |  21.29 $\pm$ 0.677 |  9.71 $\pm$ 0.304 |  10.12 $\pm$ 0.307 |  9.67 $\pm$ 0.319
Deep Adaptive Wavelet Network [89] | No |  71.52 $\pm$ 0.191 |  17.51 $\pm$ 0.569 |  4.63 $\pm$ 0.160 |  4.86 $\pm$ 0.134 |  4.83 $\pm$ 0.119
Classical Adversarial Training [3] | Yes |  59.83 $\pm$ 0.217 |  31.85 $\pm$ 0.502 |  29.44 $\pm$ 0.202 |  27.84 $\pm$ 0.318 |  28.03 $\pm$ 0.047
Fast Adversarial Training [62] | Yes |  62.40 $\pm$ 0.075 |  32.01 $\pm$ 0.135 |  26.23 $\pm$ 0.175 |  25.92 $\pm$ 0.020 |  26.22 $\pm$ 0.085
Free Adversarial Training [63] | Yes |  60.65 $\pm$ 0.211 |  31.21 $\pm$ 0.411 |  27.01 $\pm$ 0.092 |  26.54 $\pm$ 0.330 |  26.47 $\pm$ 0.224
Adt-Exp [21] | Yes |  63.80 $\pm$ 0.324 |  34.43 $\pm$ 0.430 |  27.97 $\pm$ 0.132 |  28.75 $\pm$ 0.132 |  28.40 $\pm$ 0.175
Adt-Expam [21] | Yes |  63.68 $\pm$ 0.212 |  36.33 $\pm$ 0.350 |  29.02 $\pm$ 0.159 |  29.66 $\pm$ 0.160 |  28.99 $\pm$ 0.171
Adt-Impam [21] | Yes |  63.71 $\pm$ 0.271 |  38.10 $\pm$ 0.151 |  28.25 $\pm$ 0.335 |  28.59 $\pm$ 0.311 |  28.39 $\pm$ 0.435
Adversarial Wavelet Training (ours) | Yes |  61.47 $\pm$ 0.416 |  34.80 $\pm$ 0.394 |  32.19 $\pm$ 0.166 |  30.56 $\pm$ 0.135 |  30.47 $\pm$ 0.142

We compare our proposed model with several different models. On CIFAR-10, four
models are in the natural training settings: vanilla wide residual network
(Vanilla WideResNet) [67], wavelet convolutional neural network (WCNN) [88],
scattering network [86], and deep adaptive wavelet network [89]. Other eight
methods are trained with the WideResNet models in the adversarial training
settings: Classical Adversarial Training [3], Fast Adversarial Training [62],
Free Adversarial Training [63], adversarial training with the surrogate loss
function (TRADES, regularization coefficient $\beta=6.0$) [17], friendly
adversarial training (FAT) [19], adversarial training trick method [20], the
method that the explicit adversarial distribution is defined by the
distribution transformation with activation function (Adt-Exp) [21], the
method that the explicit adversarial distribution is generated by the neural
network (Adt-Expam) [21], the method that the implicit adversarial
distribution is generated by the neural network (Adt-Impam) [21]. The models
derived from the work of adversarial distributional training [21] are the
WideResNets with a width size of 10 and a depth of 28 layers. Other
adversarial training models are WideResNets with a width of 10 and a depth of
34 layers (WideResNet-34-10). Vanilla WideResNet is in the same structure
setting as these adversarial training methods. The Fast Gradient Sign Method
(FGSM) [2], Projected Gradient PGD method [3], Momentum Iteration Method (MIM)
[4], and C&W method [5] are utilized to validate the robustness with the
perturbation budget $\varepsilon=0.031$. For multiple-step attack methods
(PGD, MIM, C&W), the perturbed number of steps is 20, and the perturbation
step size is 2. The decay factor of the MIM attack is $1.0$. The PGD attack
and adaptive C&W attack would be restarted after $20$ steps. All the multiple-
step adversarial examples are generated with random initialization. When the
perturbation budget $\varepsilon$ is 0, the clean accuracy (Acc) can be seen
as the generalization ability under the natural testing setting. The
experiment results on CIFAR-10 is illustrated in Table I. On the dataset of
CIFAR-100, similar methods are compared. The result on CIFAR-100 is
demonstrated in Table II.

As Table I illustrates, most natural training methods have high clean accuracy
on the CIFAR-10 dataset. It is accepted that adversarially-trained models
degrade a little on the metrics of clean accuracy due to discarding some model
weights unrelated to the robustness [64]. Our wavelet regularization method of
Wavelet Average Pooling helps boost the robustness and enhance adversarial
training. The evaluation result on CIFAR-100 can be seen from Table II. On the
CIFAR-100 dataset, our model also realizes considerable adversarial
robustness. Although our defense model does not perform best on the simpler
single-step FGSM attack (compared with the adversarial distributional training
method [21]), the robust accuracy metrics of our proposed method on other
multi-step adversarial attacks are better.

To prove the interpretability of the learning features of our proposed models,
the Grad-CAM [100] output on the feature heat map before fully connected
layers would be visualized. As Fig. 7 illustrates, the learnable features of
our proposed models on the adversarial image of the ostrich are best. The
previous two methods ignore the legs of the ostrich. In contrast, the heatmap
generated by our proposed method would focus on the meaningful part of the
ostrich. It seems that our model learns interpretable features of ostrich.
Nevertheless, the feature heat maps derived from the vanilla WideResNet model
[67] and the model trained by the adversarial training trick method [20] are
on unrelated regions.

Figure 7: Grad-CAM visualization of the heat map of the features on the
adversarial sample image from the CIFAR-10 dataset. Figure. (a) illustrates an
adversarial page of ostrich, which is generated by PGD attack. Figure. (b)
illustrates the extracted feature heat map of vanilla WideResNet model [67].
Figure. (c) illustrates the extracted feature heat map of the model trained by
the adversarial training trick method [20]. Figure. (d) illustrates the
extracted feature heat map of our proposed model.

###  V-C Defense Experiment Against Black-Box Attacks

The performance of our proposed method under the black-box attacks is also
evaluated. The utilized attack method is the black-box attack method based on
the natural evolutionary strategies (NES attack) [38]. The perturbation budget
of NES attack satisfies $\varepsilon=0.05$. The coefficient of gradient
estimation satisfies $fd\\_eta=2.55$. The query number cannot surpass $10000$.
The learning rate is $2.55$. The attack failure rate (AFR) is the metric of
robustness to evaluation the ratio of successful defense. The misclassified
examples in the natural settings without adversarial perturbations would not
be taken into considerations in the black-box attack experiment. The average
query number (AQN) can measure the attack efficiency. The experiment is
evaluated on the CIFAR-10 dataset. Five list methods in Table III are tested.

TABLE III: PERFORMANCE OF DIFFERENT DEFENSE METHODS AGAINST NES ATTACKS ON CIFAR-10 ($\varepsilon=0.05$). | AFR(%) | AQN
---|---|---
Classical Adversarial Training [3] |  79.40 $\pm$ 3.816 |  676 $\pm$ 55
Fast Adversarial Training [62] |  80.69 $\pm$ 1.313 |  669 $\pm$ 31
Free Adversarial Training [63] |  80.80 $\pm$ 2.697 |  694 $\pm$ 18
Adversarial Training Trick [20] |  76.94 $\pm$ 2.273 |  579 $\pm$ 13
Adversarial Wavelet Training (ours) |  82.79 $\pm$ 3.089 |  605 $\pm$ 19

NES attack [38] is based on the strategy of gradient estimation in which the
adversary cannot obtain the information of model structure. As Table III
illustrates, both AFR and AQN in the statistical test have a large standard
deviation due to the instability of black-box attacks. In such a scenario, our
proposed method could realize considerable adversarial robustness among the
compared methods under the NES attacks.

###  V-D Defense Experiment Against Adaptive Attacks

An effective defense method should avoid the phenomenon of obfuscated
gradient. Therefore, besides C&W attacks [5], the evaluation under the attacks
of Backward Pass Differentiable Approximation (BPDA) [41] would help avoid a
false sense of security. The learning rate of BPDA attack is $0.5$, and the
maximum iteration number is $100$. The robust accuracy under the range of
perturbation budgets $[0,\ 0.031,\ 0.062,\ 0.093,\ 0.124,\ 0.155]$ would be
evaluated on the CIFAR-10 dataset.

Figure 8: The robust accuracy of different defense methods under the BPDA
attacks [41].

As can be seen from the Fig. 8, the performance of our method is close to the
performance of the adversarial training trick method [20]. When the
perturbation budget increases from $0$ to $0.155$, the robust accuracy of some
other defense methods [3, 62, 63] drops near 40%. In the settings with the
perturbation budget $\varepsilon=0.155$, the robust accuracy of our method
against BPDA attack is 47.19%. Our method would not be troubled by the
phenomenon of obfuscated gradient.

###  V-E Defense Experiment Against Auto Attacks

Due to the sub-optimal step size and problems of the objective function
existed in the PGD attacks, the auto attack methods [43] can be seen as a
necessary complement of robustness evaluation. Two attack methods described in
the seminal work [43] would be evaluated, including the size-free PGD attack
method based on the cross-entropy loss (APGD-CE) and size-free PGD attack
method based on the targeted difference of logits ratio (DLR) loss (APGD-T).

TABLE IV: THE RECOGNITION PERFORMANCE (%) OF DIFFERENT DEFENSE METHODS AGAINST AUTO ATTACKS ON CIFAR-10 ($\varepsilon=0.031$). | APGD-CE Acc | APGD-T Acc
---|---|---
Classical Adversarial Training [3] |  49.61 $\pm$ 0.323 |  46.04 $\pm$ 0.253
Fast Adversarial Training [62] |  46.49 $\pm$ 0.141 |  43.53 $\pm$ 0.132
Free Adversarial Training [63] |  48.96 $\pm$ 0.164 |  46.41 $\pm$ 0.105
Adversarial Training Trick [20] |  53.63 $\pm$ 0.724 |  51.55 $\pm$ 0.239
Adversarial Wavelet Training (ours) |  54.10 $\pm$ 0.138 |  51.77 $\pm$ 0.131
TABLE V: THE RECOGNITION PERFORMANCE (%) OF DIFFERENT DEFENSE METHODS AGAINST AUTO ATTACKS ON CIFAR-100 ($\varepsilon=0.031$). | APGD-CE Acc | APGD-T Acc
---|---|---
Classical Adversarial Training [3] |  28.90 $\pm$ 0.200 |  25.77 $\pm$ 0.073
Fast Adversarial Training [62] |  25.54 $\pm$ 0.245 |  23.23 $\pm$ 0.130
Free Adversarial Training [63] |  26.53 $\pm$ 0.090 |  24.21 $\pm$ 0.128
Adversarial Wavelet Training (ours) |  31.56 $\pm$ 0.176 |  28.11 $\pm$ 0.133

The robustness evaluation result illustrated in Table IV (CIFAR-10) and Table
V (CIFAR-100) shows the considerable performance of our methods. The
evaluation pipeline of auto attack further validates the feasible robustness
of our proposed method.

###  V-F Ablation Study

Figure 9: Robustness study of wavelet regularization module
($\varepsilon=0.031$). Fig. (a) illustrates the effectiveness with/without
wavelet regularization module on ResNet [97]. Fig. (b) illustrates the
effectiveness with/without wavelet regularization module on WideResNet-34-10
[67].

To further study the applicability of our method, the ablation study is
implemented. The intuitive result is shown in Fig. 9. Adding the wavelet
regularization module to ResNet would cause the accuracy change with the value
of +0.48%, -0.76%, +0.2%, and -0.07% under the FGSM, PGD, MIM, and C&W
attacks. As a comparison, the combination of WideResNet and wavelet
regularization in the adversarial-training settings gains the increase of
robust accuracy with the value of +0.6%, +0.08%, +0.51%, and +0.28% under the
FGSM, PGD, MIM, and C&W attacks. It can be seen that our proposed method is
more applicable for the wide and deep neural network model.

###  V-G Position Of Wavelet Regularization Module

Figure 10: The influence of robustness with different deployed positions of
wavelet regularization on WideResNet-34-10 [67] ($\varepsilon=0.031$). The
experimental result of the deployment after first $3\times 3$ conv block is
marked in green color. The experimental result of the deployment before ReLU
activation function is marked in pink color. The experimental result of the
deployment after ReLU activation function is marked in blue color.

What is the effect if the wavelet regularization model is inserted in
different positions of the neural network model? To answer such a question, an
experiment is implemented. The wavelet regularization module would be inserted
after first $3\times 3$ convolutional block (conv block), before ReLU
activation function, after ReLU activation function (default setting). Fig. 10
illustrates the quantitative result. The result is:
FGSM Acc: $62.02\%\ (blue)\textgreater 61.09\%\ (pink)\textgreater
59.13\%(green)$.
PGD Acc: $55.06\%\ (pink)\textgreater 54.89\%\ (blue)\textgreater
52.83\%(green)$.
MIM Acc: $55.34\%\ (blue)\textgreater 54.78\%\ (pink)\textgreater
52.71\%(green)$.
C&W Acc: $54.29\%\ (blue)\textgreater 53.44\%\ (pink)\textgreater
51.86\%(green)$.

In a deep neural network, the shallow convolutional block would focus on the
information of edge and texture, while the deep convolutional block would
focus on the detailed information of the object part. The insertion of wavelet
regularization after the deep convolutional block could realize a positive
catalytic effect. The ReLU activation function adds nonlinearity to the deep
neural network model. The empirical result shows a tendency that adding the
wavelet regularization after the ReLU function looks more feasible.

###  V-H Influence Of Model Width And Model Depth On Adversarial Robustness

Figure 11: Robustness study of model in different widths and depths with our
proposed method. It includes ResNet34 (85M parameters), WideResNet-34-10 (193M
parameters), WideResNet-34-20 (772M parameters), and WideResNet-70-16 (1067M
parameters). The perturbation budget range is $[0,\ 0.0155,\ 0.031,\ 0.0465]$.
Fig. (a) illustrates the robustness of different models under the FGSM attack
[2]. Fig. (b) illustrates the robustness of different models under the PGD
attack [3]. Fig. (c) illustrates the robustness of different models under the
MIM attack [4]. Fig. (d) illustrates the robustness of different models under
the C&W attack [5]. Fig. (e) illustrates the robustness of different models
under the APGD-CE attack [43]. Fig. (f) illustrates the robustness of
different models under the APGD-T attack [43].

In this study, there exists a postulation that the deep wide neural network
would boost adversarial robustness. To validate such an assumption of
overparameterization, an empirical study is made to explore the influence of
model width and model depth on adversarial robustness. All listed four models
(ResNet34, WideResNet-34-10, WideResNet-34-20, and WideResNet-70-16) are
combined with the module of wavelet regularization.

Fig.11 illustrates the quantitative result of the experiment for exploration
of model-width and model-depth on adversarial robustness. In the overall
dimension, WideResNet-70-16 is most robust under adversarial attacks. It is
followed by WideResNet-34-20. WideResNet-34-10 surpasses ResNet34 except some
metric under the large perturbation ($\varepsilon=0.0465$). To sum up, the
combination of our regularization method and overparameterized model is
beneficial to adversarial robustness.

###  V-I Robustness Evaluation On The Frequency Perspective

In the previous work of the F-Principle of deep learning, Dong et al. [27]
argued that discarding the high-frequency elements absolutely would have a
negative impact on the robustness. Adversarial training just biases the
perturbations towards the low-frequency domain. The Fourier heat map is
proposed in this work [27] to evaluate the robustness in the frequency domain.
In our experiment, the frequency perturbations are added to the CIFAR-10
validation dataset. The robustness of models on the Fourier perspective in
different training settings is evaluated intuitively. The Fourier heat map can
give a visualization of the robustness of different models in the frequency
domain. The model sensitivity to perturbations aligns with different Fourier
basis vectors on the CIFAR-10 validation dataset. As Fig. 12 shows, the
WideResNet model [67] of natural training is the most fragile which shows
higher concentrations in the high-frequency domain and has the largest error
rate. In contrast, the model of adversarial training trick method [20]
performs better than the natural-training model. However, our proposed method
performs best on the metrics of test error. Although the model trained by the
adversarial training trick method [20] lifts the robustness under the high-
frequency perturbations, it sacrifices clean accuracy and robustness under the
low-frequency perturbations. Our proposed model has a considerable robustness
performance under both low-frequency perturbations and high-frequency
perturbations.

Figure 12: Fourier heat map of test error on different defense models
($\varepsilon=0.031$). Fig. (a) illustrates the heat map of the vanilla
WideResNet model [67]. Fig. (b) illustrates the heat map of the adversarial
training trick method [20]. Fig. (c) illustrates the heat map of our
WideWaveletResNet Model.

###  V-J Comparison Experiment On Various Wavelet Bases

Besides the Haar wavelet base, we also test the wavelet regularizations with
other base functions: Daubechies-5 wavelet base (Db5), Biorthogonal-3.1
wavelet base (bior3.1), Coiflets 4 wavelet base (Coif4), Discrete Meyer
wavelet base (Dmey), Reverse Biorthogonal-2.2 wavelet base (Rbio2.2), and
Symlets 4 wavelet base (Sym4). The analysis of these wavelet base functions
can be seen in the mathematical books [33, 82, 32]. The attack setting is the
same as the main experiment with the perturbation budget $\varepsilon=0.031$
and step sizes $20$ for PGD attack, MIM attack, and C&W attack. Also the auto
attacks including the APGD-CE attack and the APGD-T attack are tested.

TABLE VI: THE EVALUATION METRICS (%) OF WAVELET REGULARIZATION ON DIFFERENT BASE FUNCTIONS UNDER WHITE-BOX ATTACKS AND AUTO-ATTACKS ON CIFAR-10 ($\varepsilon=0.031$). | Clean Acc | FGSM Acc | PGD Acc | MIM Acc | C&W Acc | APGD-CE Acc | APGD-T Acc
---|---|---|---|---|---|---|---
Db5 |  86.72 $\pm$ 0.103 |  61.19 $\pm$ 0.155 |  54.87 $\pm$ 0.234 |  54.74 $\pm$ 0.217 |  54.05 $\pm$ 0.186 |  54.26 $\pm$ 0.204 |  51.79 $\pm$ 0.217
Bior3.1 |  86.56 $\pm$ 0.206 |  61.14 $\pm$ 0.138 |  54.93 $\pm$ 0.096 |  54.66 $\pm$ 0.161 |  54.03 $\pm$ 0.143 |  54.24 $\pm$ 0.107 |  51.61 $\pm$ 0.237
Coif4 |  86.72 $\pm$ 0.482 |  61.36 $\pm$ 0.141 |  55.25 $\pm$ 0.249 |  54.83 $\pm$ 0.238 |  54.50 $\pm$ 0.083 |  54.64 $\pm$ 0.195 |  52.23 $\pm$ 0.131
Dmey |  86.75 $\pm$ 0.293 |  61.14 $\pm$ 0.250 |  54.82 $\pm$ 0.141 |  54.49 $\pm$ 0.081 |  54.10 $\pm$ 0.221 |  54.17 $\pm$ 0.146 |  51.64 $\pm$ 0.345
Rbio2.2 |  87.07 $\pm$ 0.319 |  61.51 $\pm$ 0.396 |  54.70 $\pm$ 0.331 |  54.53 $\pm$ 0.267 |  53.88 $\pm$ 0.154 |  54.00 $\pm$ 0.301 |  51.44 $\pm$ 0.148
Sym4 |  86.59 $\pm$ 0.111 |  61.18 $\pm$ 0.214 |  55.26 $\pm$ 0.209 |  54.90 $\pm$ 0.137 |  54.25 $\pm$ 0.086 |  54.57 $\pm$ 0.228 |  52.03 $\pm$ 0.164
Haar |  87.34 $\pm$ 0.085 |  61.61 $\pm$ 0.361 |  54.76 $\pm$ 0.116 |  54.47 $\pm$ 0.602 |  54.25 $\pm$ 0.074 |  54.10 $\pm$ 0.138 |  51.77 $\pm$ 0.131
TABLE VII: THE EVALUATION METRICS (%) OF WAVELET REGULARIZATION ON DIFFERENT BASE FUNCTIONS UNDER WHITE-BOX ATTACKS AND AUTO-ATTACKS ON CIFAR-100 ($\varepsilon=0.031$). | Clean Acc | FGSM Acc | PGD Acc | MIM Acc | C&W Acc | APGD-CE Acc | APGD-T Acc
---|---|---|---|---|---|---|---
Db5 |  61.08 $\pm$ 0.801 |  34.80 $\pm$ 0.339 |  32.01 $\pm$ 0.261 |  30.44 $\pm$ 0.298 |  30.73 $\pm$ 0.196 |  31.27 $\pm$ 0.321 |  28.21 $\pm$ 0.170
Bior3.1 |  60.88 $\pm$ 0.133 |  34.68 $\pm$ 0.059 |  32.14 $\pm$ 0.189 |  30.46 $\pm$ 0.112 |  30.55 $\pm$ 0.244 |  31.52 $\pm$ 0.166 |  28.19 $\pm$ 0.136
Coif4 |  60.33 $\pm$ 0.200 |  34.48 $\pm$ 0.203 |  32.03 $\pm$ 0.307 |  30.53 $\pm$ 0.164 |  30.37 $\pm$ 0.316 |  31.88 $\pm$ 0.270 |  27.89 $\pm$ 0.307
Dmey |  61.20 $\pm$ 0.388 |  34.60 $\pm$ 0.243 |  32.02 $\pm$ 0.267 |  30.50 $\pm$ 0.304 |  30.61 $\pm$ 0.126 |  31.40 $\pm$ 0.314 |  28.23 $\pm$ 0.128
Rbio2.2 |  61.42 $\pm$ 0.480 |  34.69 $\pm$ 0.353 |  32.00 $\pm$ 0.208 |  30.50 $\pm$ 0.126 |  30.49 $\pm$ 0.296 |  31.43 $\pm$ 0.237 |  28.08 $\pm$ 0.252
Sym4 |  61.35 $\pm$ 0.374 |  34.49 $\pm$ 0.381 |  31.95 $\pm$ 0.135 |  30.41 $\pm$ 0.218 |  30.37 $\pm$ 0.310 |  31.21 $\pm$ 0.315 |  27.93 $\pm$ 0.304
Haar |  61.47 $\pm$ 0.416 |  34.80 $\pm$ 0.394 |  32.19 $\pm$ 0.166 |  30.56 $\pm$ 0.135 |  30.47 $\pm$ 0.142 |  31.56 $\pm$ 0.176 |  28.11 $\pm$ 0.133

As Table VI and Table VII exhibit, the Haar base function is the most
suitable. On CIFAR-10, the Sym4 base performs best under the PGD attack and
MIM attack, and the Coif4 base performs best under the adaptive C&W attack and
auto attacks. Nevertheless, the Haar base performs best without adversarial
perturbation and under the FGSM attack. On CIFAR-100, the Haar base performs
best on several metrics (clean Acc, FGSM Acc, PGD Acc, MIM Acc). Summarized
from the numerical results on the two datasets, the utilization of the Haar
function would be more feasible at the comprehensive level. As we recall from
Table VIII, the Haar function is orthogonal, symmetrical, and compactly
supported. It has a short support length, a short filter size, and a short
vanishing movement (written as “Movement” in the table). The goal of wavelet
decomposition is to realize the sparsity and reduce the structural risk for
robustness improvement. The empirical result validates that the Haar function
with a short vanishing movement adapts to such a scenario.

TABLE VIII: THE PROPERTY COMPARISON OF DIFFERENT WAVELET BASES. | Orthogonality | Double orthogonality | Compactness | Support size | Filter size | Symmetry | Movement
---|---|---|---|---|---|---|---
Haar | Yes | Yes | Yes | 1 | 2 | Yes | 1
Db5 | Yes | Yes | Yes | 9 | 10 | Approximately yes | 5
Bior3.1 | No | Yes | Yes | 3 | 8 | No | 2
Coif4 | Yes | Yes | Yes | 23 | 24 | Approximately yes | 8
Dmey | No | No | No | N/A | N/A | Yes | N/A
Rbio2.2 | No | Yes | Yes | N/A | N/A | Yes | N/A
Sym4 | Yes | Yes | Yes | 7 | 8 | Approximately yes | 4

###  V-K Robustness Evaluation On Two Aggregation Strategies

TABLE IX: THE PERFORMANCE (%) OF DIFFERENT WAVELET AGGREGATION STRATEGIES ON CIFAR-10 AND CIFAR-100 ($\varepsilon=0.031$). | Method | Clean Acc | FGSM Acc | PGD Acc | MIM Acc | C&W Acc | APGD-CE Acc | APGD-T Acc
---|---|---|---|---|---|---|---|---
CIFAR-10 | LPF |  86.94 $\pm$ 0.151 |  61.11 $\pm$ 0.201 |  54.94 $\pm$ 0.121 |  54.57 $\pm$ 0.110 |  53.55 $\pm$ 0.177 |  54.14 $\pm$ 0.111 |  51.28 $\pm$ 0.115
WAP |  87.34 $\pm$ 0.085 |  61.61 $\pm$ 0.361 |  54.76 $\pm$ 0.116 |  54.47 $\pm$ 0.602 |  54.25 $\pm$ 0.074 |  54.10 $\pm$ 0.138 |  51.77 $\pm$ 0.131
CIFAR-100 | LPF |  61.66 $\pm$ 0.489 |  34.36 $\pm$ 0.721 |  31.31 $\pm$ 0.948 |  30.03 $\pm$ 0.794 |  29.77 $\pm$ 0.675 |  30.69 $\pm$ 0.942 |  27.48 $\pm$ 0.720
WAP |  61.47 $\pm$ 0.416 |  34.80 $\pm$ 0.394 |  32.19 $\pm$ 0.166 |  30.56 $\pm$ 0.135 |  30.47 $\pm$ 0.142 |  31.56 $\pm$ 0.176 |  28.11 $\pm$ 0.133

We compare our proposed method of Wavelet Average Pooling (WAP) with the
strategy of low-pass filter (LPF) which means that only the low-frequency
elements with the approximation would be retained. As Table IX illustrates,
the aggregation strategy of Wavelet Average Pooling realizes a better trade-
off of clean accuracy and adversarial robustness. On CIFAR-10, the method of
WAP surpasses the method of LPF on the metrics of clean Acc, FGSM Acc,
adaptive C&W Acc, and APGD-T Acc. Nevertheless, the method of WAP is inferior
to the method of LPF on the metrics of PGD Acc, MIM Acc, and APGD-CE Acc. On
CIFAR-100, the method of WAP surpass the method of LPF under all types of
adversarial perturbations. On CIFAR-100, the clean Acc of LPF method is better
than the clean Acc of WAP method. The reason can be attributed to the
F-Principle, the low-frequency elements are related to the generalization
ability while the high-frequency elements have tight connections to the
robustness [27]. Filtering the high-frequency elements may obtain a higher
clean Acc on the CIFAR-100 dataset. However, the trade-off between the low-
frequency elements and the high-frequency elements is more applicable to
realize adversarial robustness.

###  V-L Discussion

The proposal of the wavelet regularization of adversarial training aims to
improve the adversarial robustness of the neural networks in the frequency
domain. We explore the mathematical property of wavelet transform and
postulate that its regularization character can help realize a small Lipschitz
constraint which is good for robustness. The training record of the experiment
shows that our proposed method’s gradient value is lower than $0.05$ which
means that our wavelet regularization method enhances adversarial training
with a low Lipschitz constraint. The empirical study of the defense against
white-box attacks, black-box attacks, adaptive attacks, and auto attacks
validates the effectiveness of our proposed method. The intuitive
visualization experiment shows that our proposed method could realize
interpretability in the process of feature extraction. Furthermore, the
ablation study and the comparison experiment with the wavelet regularization
method on various models such as ResNet34, WideResNet-34-10, WideResNet-34-20,
and WideResNet-76-10 demonstrates that our proposed method could boost the
adversarial robustness on the basis of overparameterization. The lifting is
more intuitive in a wide and deep neural network.

Our method is inspired by the F-Principle. In the research of deep learning,
the F-Principle expresses an opinion that the low-frequency elements are
related to the generalization of natural training while high-frequency
elements are related to the robustness under the perturbations. We utilize the
Fourier heat map [27] to validate our robustness in the frequency domain. By
biasing the high frequency towards the low frequency, robustness can be
guaranteed with a low error rate. In the sub-experiment of different
aggregation strategies of wavelets, we find that discarding the high-frequency
elements absolutely leads to the degradation of robustness compared to Wavelet
Average Pooling. In summary, our adversarial wavelet training method can be
regarded as a positive example of the F-Principle.

In the family of wavelet bases, the Haar base is orthogonal, doubly
orthogonal, and compactly supported. It has a short filter size, a short
support size, and a short vanish movement. This character leads to a robust
sparsity property. The utilization of the Haar function cannot guarantee that
it is always the most robust under all adversarial attacks. However, it is
most suitable to choose the Haar function as a default setting of wavelet
regularization at a comprehensive level.

##  VI Conclusion

Adversarial vulnerability can be regarded as a high-frequency phenomenon.
Inspired by such a motivation, we propose a wavelet regularization method and
integrate it into a wide residual neural network to form a WideWaveletResNet.
During the process of adversarial training, our wavelet regularization method
based on the Haar function could realize a small Lipschitz constraint for the
guarantee of adversarial robustness. Our proposed model achieves considerable
robust performance under the diverse types of attacks on the datasets of
CIFAR-10 and CIFAR-100. The enhancement of adversarial robustness is intuitive
when the wavelet regularization method works on a wide and deep neural
network. The visualization experiment result also validates the
interpretability of our model’s learning features and the robustness of our
model from a frequency perspective. A detailed comparison experiment is
implemented to show the feasibility of the Haar base function. The absence of
the evaluation of the proposed method on the high-resolution database is
perhaps a limitation of current research work. In the future, we would extend
our wavelet regularization method to high-resolution image datasets to make
our methods more applicable. Moreover, the relationship between SRM in
traditional statistical learning and overparameterization in deep learning
deserves in-depth research. Besides the improvement of adversarial robustness
from the perspective of model structure, the utilization of robust non-convex
optimization methods and domain adaptation methods [101, 102] is also feasible
to boost adversarial robustness.

## Acknowledgment

This work was supported by the National Natural Science Foundation of China
under Grant No. 61701348. The authors would like to thank TUEV SUED for the
kind and generous support. The team of the Sino-German Center of Intelligent
Systems in Tongji University gives us support to carry our research. The
authors are grateful to the Ph.D. candidate of Tongji University, Mr. Yujian
Mo for his suggestion on our research. The authors would appreciate Dr. Qi
Deng and Dr. Yunmei Shi for their suggestions on our manuscript. We sincerely
thank Mr. Xi Fang and Mr. Huan Hua for their help in our experimental work.

## References

  * [1] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in _ICLR_ , 2014\.
  * [2] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in _ICLR_ , 2015.
  * [3] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _ICLR_ , 2018.
  * [4] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boosting adversarial attacks with momentum,” in _CVPR_ , 2018, pp. 9186–9193.
  * [5] N. Carlini and D. A. Wagner, “Towards evaluating the robustness of neural networks,” in _IEEE Symposium on Security and Privacy (SP)_ , 2017, pp. 39–57.
  * [6] P. Chen, H. Zhang, Y. Sharma, J. Yi, and C. Hsieh, “ZOO: zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” in _AISec Workshop@CCS_ , 2017, pp. 15–26.
  * [7] C. Tu, P. Ting, P. Chen, S. Liu, H. Zhang, J. Yi, C. Hsieh, and S. Cheng, “Autozoom: Autoencoder-based zeroth order optimization method for attacking black-box neural networks,” in _AAAI_ , 2019, pp. 742–749.
  * [8] Y. Dong, T. Pang, H. Su, and J. Zhu, “Evading defenses to transferable adversarial examples by translation-invariant attacks,” in _CVPR_ , 2019, pp. 4312–4321.
  * [9] Y. Li, L. Li, L. Wang, T. Zhang, and B. Gong, “NATTACK: learning the distributions of adversarial examples for an improved black-box attack on deep neural networks,” in _ICML_ , 2019, pp. 3866–3876.
  * [10] A. Ilyas, L. Engstrom, and A. Madry, “Prior convictions: Black-box adversarial attacks with bandits and priors,” in _ICLR_ , 2019.
  * [11] N. Papernot, P. D. McDaniel, X. Wu, S. Jha, and A. Swami, “Distillation as a defense to adversarial perturbations against deep neural networks,” in _IEEE Symposium on Security and Privacy (SP)_ , 2016, pp. 582–597.
  * [12] C. Xie, J. Wang, Z. Zhang, Z. Ren, and A. L. Yuille, “Mitigating adversarial effects through randomization,” in _ICLR_ , 2018.
  * [13] F. Liao, M. Liang, Y. Dong, T. Pang, X. Hu, and J. Zhu, “Defense against adversarial attacks using high-level representation guided denoiser,” in _CVPR_ , 2018, pp. 1778–1787.
  * [14] Y. Song, T. Kim, S. Nowozin, S. Ermon, and N. Kushman, “Pixeldefend: Leveraging generative models to understand and defend against adversarial examples,” in _ICLR_ , 2018.
  * [15] A. Agarwal, M. Vatsa, R. Singh, and N. K. Ratha, “Cognitive data augmentation for adversarial defense via pixel masking,” _Pattern Recognit. Lett._ , vol. 146, pp. 244–251, 2021.
  * [16] X. Liu, Y. Li, C. Wu, and C. Hsieh, “Adv-bnn: Improved adversarial defense through robust bayesian neural network,” in _ICLR_ , 2019.
  * [17] H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan, “Theoretically principled trade-off between robustness and accuracy,” in _ICML_ , 2019, pp. 7472–7482.
  * [18] L. Rice, E. Wong, and J. Z. Kolter, “Overfitting in adversarially robust deep learning,” in _ICML_ , 2020, pp. 8093–8104.
  * [19] J. Zhang, X. Xu, B. Han, G. Niu, L. Cui, M. Sugiyama, and M. S. Kankanhalli, “Attacks which do not kill training make adversarial learning stronger,” in _ICML_ , 2020, pp. 11 278–11 287.
  * [20] T. Pang, X. Yang, Y. Dong, H. Su, and J. Zhu, “Bag of tricks for adversarial training,” in _ICLR_ , 2021.
  * [21] Y. Dong, Z. Deng, T. Pang, J. Zhu, and H. Su, “Adversarial distributional training for robust deep learning,” in _NeurIPS_ , 2020.
  * [22] A. S. Ross and F. Doshi-Velez, “Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients,” in _AAAI_ , 2018, pp. 1660–1669.
  * [23] H. Kannan, A. Kurakin, and I. J. Goodfellow, “Adversarial logit pairing,” _CoRR_ , vol. abs/1803.06373, 2018.
  * [24] Z. He, A. S. Rakin, and D. Fan, “Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack,” in _CVPR_ , 2019, pp. 588–597.
  * [25] D. Jakubovitz and R. Giryes, “Improving DNN robustness to adversarial attacks using jacobian regularization,” in _ECCV_ , 2018, pp. 525–541.
  * [26] S. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. Mann, “Data augmentation can improve robustness,” in _NeurIPS_ , 2021.
  * [27] D. Yin, R. G. Lopes, J. Shlens, E. D. Cubuk, and J. Gilmer, “A fourier perspective on model robustness in computer vision,” in _NeurIPS_ , 2019, pp. 13 255–13 265.
  * [28] C. Zhang, P. Benz, A. Karjauv, and I. S. Kweon, “Universal adversarial perturbations through the lens of deep steganography: Towards a fourier perspective,” in _AAAI_ , 2021, pp. 3296–3304.
  * [29] Z. J. Xu, Y. Zhang, T. Luo, Y. Xiao, and Z. Ma, “Frequency principle: Fourier analysis sheds light on deep neural networks,” _Commun. Comput. Phys._ , vol. 28, no. 5, pp. 1746–1767, 2020.
  * [30] T. Luo, Z. Ma, Z. J. Xu, and Y. Zhang, “Theory of the frequency principle for general deep neural networks,” _CSIAM Trans. Appl. Math._ , vol. 2, no. 3, pp. 484–507, 2021.
  * [31] H. Wang, X. Wu, Z. Huang, and E. P. Xing, “High-frequency component helps explain the generalization of convolutional neural networks,” in _CVPR_ , 2020, pp. 8681–8691.
  * [32] S. Mallat, _A Wavelet Tour of Signal Processing - The Sparse Way, 3rd Edition_. Academic Press, 2009.
  * [33] I. Daubechies, _Ten Lectures on Wavelets_. SIAM, 1992.
  * [34] O. Knagg, “Know your enemy: How you can create and defend against adversarial attacks,” https://towardsdatascience.com/know-your-enemy-7f7c5038bdf3, 2019\.
  * [35] A. Krizhevsky and G. Hinton, “A learning multiple layers of features from tiny images,” http://www.cs.toronto.edu/~kriz/cifar.html, 2009.
  * [36] S. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: A simple and accurate method to fool deep neural networks,” in _CVPR_ , 2016, pp. 2574–2582.
  * [37] Y. Shi, S. Wang, and Y. Han, “Curls & whey: Boosting black-box adversarial attacks,” in _CVPR_ , 2019, pp. 6519–6527.
  * [38] A. Ilyas, L. Engstrom, A. Athalye, and J. Lin, “Black-box adversarial attacks with limited queries and information,” in _ICML_ , 2018, pp. 2142–2151.
  * [39] W. Brendel, J. Rauber, and M. Bethge, “Decision-based adversarial attacks: Reliable attacks against black-box machine learning models,” in _ICLR_ , 2018\.
  * [40] W. Chen, Z. Zhang, X. Hu, and B. Wu, “Boosting decision-based black-box adversarial attacks with random sign flip,” in _ECCV_ , 2020, pp. 276–293.
  * [41] A. Athalye, N. Carlini, and D. A. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” in _ICML_ , 2018, pp. 274–283.
  * [42] F. Tramèr, N. Carlini, W. Brendel, and A. Madry, “On adaptive attacks to adversarial example defenses,” in _NeurIPS_ , 2020.
  * [43] F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” in _ICML_ , 2020, pp. 2206–2216.
  * [44] J. Lu, T. Issaranon, and D. A. Forsyth, “Safetynet: Detecting and rejecting adversarial examples robustly,” in _ICCV_ , 2017, pp. 446–454.
  * [45] A. Agarwal, R. Singh, M. Vatsa, and N. K. Ratha, “Image transformation-based defense against adversarial perturbation on deep learning models,” _IEEE Trans. Dependable Secur. Comput._ , vol. 18, no. 5, pp. 2106–2121, 2021.
  * [46] Y. Li, J. Bradshaw, and Y. Sharma, “Are generative classifiers more robust to adversarial attacks?” in _ICML_ , 2019, pp. 3804–3814.
  * [47] P. Ghosh, A. Losalka, and M. J. Black, “Resisting adversarial attacks using gaussian mixture variational autoencoders,” in _AAAI_ , 2019, pp. 541–548.
  * [48] N. Akhtar, J. Liu, and A. Mian, “Defense against universal adversarial perturbations,” in _CVPR_ , 2018, pp. 3389–3398.
  * [49] X. Ma, B. Li, Y. Wang, S. M. Erfani, S. N. R. Wijewickrema, G. Schoenebeck, D. Song, M. E. Houle, and J. Bailey, “Characterizing adversarial subspaces using local intrinsic dimensionality,” in _ICLR_ , 2018.
  * [50] T. Pang, C. Du, Y. Dong, and J. Zhu, “Towards robust detection of adversarial examples,” in _NeurIPS_ , 2018, pp. 4584–4594.
  * [51] C. Zhao, P. T. Fletcher, M. Yu, Y. Peng, G. Zhang, and C. Shen, “The adversarial attack and detection under the fisher information metric,” in _AAAI_ , 2019, pp. 5869–5876.
  * [52] A. Agarwal, G. Goswami, M. Vatsa, R. Singh, and N. K. Ratha, “Damad: Database, attack, and model agnostic adversarial perturbation detector,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–13, 2021.
  * [53] N. Das, M. Shanbhogue, S. Chen, F. Hohman, S. Li, L. Chen, M. E. Kounavis, and D. H. Chau, “SHIELD: fast, practical defense and vaccination for deep learning using JPEG compression,” in _KDD_ , 2018, pp. 196–204.
  * [54] C. Guo, M. Rana, M. Cissé, and L. van der Maaten, “Countering adversarial images using input transformations,” in _ICLR_ , 2018.
  * [55] W. Xu, D. Evans, and Y. Qi, “Feature squeezing: Detecting adversarial examples in deep neural networks,” in _NDSS_ , 2018.
  * [56] H. Yang, J. Zhang, H. Dong, N. Inkawhich, A. Gardner, A. Touchet, W. Wilkes, H. Berry, and H. Li, “DVERGE: diversifying vulnerabilities for enhanced robust generation of ensembles,” in _NeurIPS_ , 2020.
  * [57] T. Pang, K. Xu, C. Du, N. Chen, and J. Zhu, “Improving adversarial robustness via promoting ensemble diversity,” in _ICML_ , 2019, pp. 4970–4979.
  * [58] X. Liu, M. Cheng, H. Zhang, and C. Hsieh, “Towards robust neural networks via random self-ensemble,” in _ECCV_ , 2018, pp. 381–397.
  * [59] Y. Guo, C. Zhang, C. Zhang, and Y. Chen, “Sparse dnns with improved adversarial robustness,” in _NeurIPS_ , 2018, pp. 240–249.
  * [60] F. Tramèr, A. Kurakin, N. Papernot, I. J. Goodfellow, D. Boneh, and P. D. McDaniel, “Ensemble adversarial training: Attacks and defenses,” in _ICLR_ , 2018.
  * [61] L. Huang, C. Zhang, and H. Zhang, “Self-adaptive training: beyond empirical risk minimization,” in _NeurIPS_ , 2020.
  * [62] E. Wong, L. Rice, and J. Z. Kolter, “Fast is better than free: Revisiting adversarial training,” in _ICLR_ , 2020.
  * [63] A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. P. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, “Adversarial training for free!” in _NeurIPS_ , 2019, pp. 3353–3364.
  * [64] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, “Robustness may be at odds with accuracy,” in _ICLR_ , 2019.
  * [65] A. Raghunathan, S. M. Xie, F. Yang, J. C. Duchi, and P. Liang, “Understanding and mitigating the tradeoff between robustness and accuracy,” in _ICML_ , 2020, pp. 7909–7919.
  * [66] Y. Yang, C. Rashtchian, H. Zhang, R. R. Salakhutdinov, and K. Chaudhuri, “A closer look at accuracy vs. robustness,” in _NeurIPS_ , 2020.
  * [67] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in _BMVC_ , 2016\.
  * [68] S. S. Du, J. D. Lee, H. Li, L. Wang, and X. Zhai, “Gradient descent finds global minima of deep neural networks,” in _ICML_ , 2019, pp. 1675–1685.
  * [69] Q. Nguyen and M. Hein, “The loss surface of deep and wide neural networks,” in _ICML_ , 2017, pp. 2603–2612.
  * [70] L. Wu, C. Ma, and W. E, “How SGD selects the global minima in over-parameterized learning: A dynamical stability perspective,” in _NeurIPS_ , 2018, pp. 8289–8298.
  * [71] A. G. de G. Matthews, J. Hron, M. Rowland, R. E. Turner, and Z. Ghahramani, “Gaussian process behaviour in wide deep neural networks,” in _ICLR_ , 2018\.
  * [72] J. Lee, L. Xiao, S. S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington, “Wide neural networks of any depth evolve as linear models under gradient descent,” in _NeurIPS_ , 2019, pp. 8570–8581.
  * [73] Y. Li and Y. Yuan, “Convergence analysis of two-layer neural networks with relu activation,” in _NeurlIPS_ , 2017, pp. 597–607.
  * [74] Y. Cao and Q. Gu, “Generalization bounds of stochastic gradient descent for wide and deep neural networks,” in _NeurIPS_ , 2019, pp. 10 835–10 845.
  * [75] Z. Allen-Zhu, Y. Li, and Z. Song, “A convergence theory for deep learning via over-parameterization,” in _ICML_ , 2019, pp. 242–252.
  * [76] D. Zou, Y. Cao, D. Zhou, and Q. Gu, “Gradient descent optimizes over-parameterized deep relu networks,” _Mach. Learn._ , vol. 109, no. 3, pp. 467–492, 2020.
  * [77] S. Bubeck, Y. Li, and D. M. Nagaraj, “A law of robustness for two-layers neural networks,” in _COLT_ , 2021, pp. 804–820.
  * [78] S. Bubeck and M. Sellke, “A universal law of robustness via isoperimetry,” in _NeurIPS_ , 2021.
  * [79] B. Wu, J. Chen, D. Cai, X. He, and Q. Gu, “Do wider neural networks really help adversarial robustness?” in _NeurIPS_ , 2021.
  * [80] S. Moosavi-Dezfooli, A. Fawzi, J. Uesato, and P. Frossard, “Robustness via curvature regularization, and vice versa,” in _CVPR_ , 2019, pp. 9078–9086.
  * [81] S. Yun, D. Han, S. Chun, S. J. Oh, Y. Yoo, and J. Choe, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in _ICCV_ , 2019, pp. 6022–6031.
  * [82] Y. Meyer, _Wavelets - algorithms & applications_. SIAM, 1993.
  * [83] M. Gavish, B. Nadler, and R. R. Coifman, “Multiscale wavelets on trees, graphs and high dimensional data: Theory and applications to semi supervised learning,” in _ICML_ , 2010, pp. 367–374.
  * [84] O. Elisha and S. Dekel, “Function space analysis of deep learning representation layers,” _CoRR_ , vol. abs/1710.03263, 2017.
  * [85] J. Bruna and S. Mallat, “Invariant scattering convolution networks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 35, no. 8, pp. 1872–1886, 2013.
  * [86] E. Oyallon, E. Belilovsky, and S. Zagoruyko, “Scaling the scattering transform: Deep hybrid networks,” in _ICCV_ , 2017, pp. 5619–5628.
  * [87] X. Cheng, X. Chen, and S. Mallat, “Deep haar scattering networks,” _CoRR_ , vol. abs/1509.09187, 2015.
  * [88] S. Fujieda, K. Takayama, and T. Hachisuka, “Wavelet convolutional neural networks,” _CoRR_ , vol. abs/1805.08620, 2018.
  * [89] M. Bastidas-Rodríguez, A. Gruson, L. F. Polanía, S. Fujieda, F. A. P. Ortíz, K. Takayama, and T. Hachisuka, “Deep adaptive wavelet network,” in _WACV_ , 2020, pp. 3100–3108.
  * [90] T. Williams and R. Li, “Wavelet pooling for convolutional neural networks,” in _ICLR_ , 2018.
  * [91] Q. Li, L. Shen, S. Guo, and Z. Lai, “Wavelet integrated cnns for noise-robust image classification,” in _CVPR_ , 2020, pp. 7243–7252.
  * [92] D. Anshumaan, A. Agarwal, M. Vatsa, and R. Singh, “Wavetransform: Crafting adversarial examples via input decomposition,” in _ECCV Workshops_ , 2020, pp. 152–168.
  * [93] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 37, no. 9, pp. 1904–1916, 2015.
  * [94] A. Haar, “Zur theorie der orthogonalen funktionen-systeme,” _Math. Ann._ , vol. 69, pp. 331–371, 1910.
  * [95] P. A. Viola and M. J. Jones, “Robust real-time face detection,” _Int. J. Comput. Vis._ , vol. 57, no. 2, pp. 137–154, 2004.
  * [96] S. Mallat, “A theory for multiresolution signal decomposition: The wavelet representation,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 11, no. 7, pp. 674–693, 1989.
  * [97] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016, pp. 770–778.
  * [98] W. E, C. Ma, and L. Wu, “The generalization error of the minimum-norm solutions for over-parameterized neural networks,” _Pure and Applied Functional Analysis_ , vol. 5, no. 6, pp. 1145–1160, 2020.
  * [99] S. Oymak and M. Soltanolkotabi, “Toward moderate overparameterization: Global convergence guarantees for training shallow neural networks,” _IEEE J. Sel. Areas Inf. Theory_ , vol. 1, no. 1, pp. 84–105, 2020.
  * [100] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” _Int. J. Comput. Vis._ , vol. 128, no. 2, pp. 336–359, 2020\.
  * [101] Q. Kang, S. Yao, M. Zhou, K. Zhang, and A. Abusorrah, “Effective visual domain adaptation via generative adversarial distribution matching,” _IEEE Trans. Neural Networks Learn. Syst._ , vol. 32, no. 9, pp. 3919–3929, 2021.
  * [102] G. Cai, Y. Wang, L. He, and M. Zhou, “Unsupervised domain adaptation with adversarial residual transform networks,” _IEEE Trans. Neural Networks Learn. Syst._ , vol. 31, no. 8, pp. 3073–3086, 2020.

|  Jun Yan received a bachelor degree from the Department of Electronic
Engineering, Shanghai Jiaotong University in 2009 and a master degree from the
Chinese-German High College, Tongji University in 2012. After the eight-year
working experience in the industrial field, he is now a Ph.D. candidate in the
Department of Information and Communication Engineering, Tongji University,
China. His research interest is the theory of adversarial machine learning and
the design of new deep learning models in autonomous vehicles.
---|---
|  Huilin Yin received the Ph.D. degree in control theory and control
engineering from Tongji University, China, in 2006. She is currently the
chaired professor of TUEV SUED Chair with the Electronic and Information
Engineering College, Tongji University. She received the M.S. double-degree
from Tongji University and the Technical University of Munich, Germany. Her
research interests include environment perception of intelligent vehicles and
safety for autonomous driving.
---|---
|  Xiaoyang Deng Deng Xiaoyang got his bachelor’s degree from the School of
Electronics and Information Engineering of Tongji University, China. He is
currently studying for a master’s degree in the Department of Control Science
and Engineering, Tongji University. He focuses on the theory of adversarial
machine learning.
---|---
|  Ziming Zhao Ziming Zhao received the bachelor degree from Tongji
University, in 2022. From the fall of 2022, he is a master candidate at the
School of Electronics and Information Engineering, Tongji University. He
focuses on the deep learning theory.
---|---
|  Wanchenge Ge is a professor in the Department of Information and
Communication Engineering, Tongji University, China. His research interest is
on wireless communication and artificial intelligence.
---|---
|  Hao Zhang received the B.Sc. degree in automatic control from the Wuhan
University of Technology, Wuhan, China, in 2001, and the Ph.D. degree in
control theory and control engineering from the Huazhong University of Science
and Technology, Wuhan, in 2007. From 2011 to 2013, she was a Postdoctoral
Fellow with the City University of Hong Kong, Hong Kong. She is currently a
Professor at the School of Electronics and Information Engineering, Tongji
University, Shanghai, China. Her research interests include network-based
control systems, multiagent systems, and UAVs. Prof. Zhang is an Associate
Editor of IEEE Intelligent Transportation Systems Magazine.
---|---
|  Gerhard Rigoll received the Dr.-Ing. habil. degree in 1991 from Stuttgart
University with a thesis on speech synthesis. In 1993 he was appointed to full
professor of computer science at Gerhard-Mercator- University in Duisburg,
Germany. In 2002, he joined Technische Universität München (TUM), where he is
now heading the institute for Human-Machine Communication. Dr. Rigoll is an
IEEE Fellow (for contributions to multimodal human-machine communication) and
is the author and co-author of more than 550 papers in the field of pattern
recognition.
---|---

##  Appendix A Proof

###  A-A Proof Of Theorem III.1

###### Proof.

Since $\int dx\psi(x)=0$, it leads to the deduction that
$\left\langle\psi^{a,b},f\right\rangle=\int
dx|a|^{-1/2}\psi\left(\frac{x-b}{a}\right)[f(x)-f(b)]$. Hence, we have an
inference to reach the conclusion of the proof.

$\displaystyle\left|\left\langle\psi^{a,b},f\right\rangle\right|$
$\displaystyle\leq\int
dx|a|^{-1/2}\left|\psi\left(\frac{x-b}{a}\right)\right|C|x-b|^{\alpha}$
$\displaystyle\leq C|a|^{\alpha+1/2}\int
dx|\psi(x+\delta)||x+\delta|^{\alpha}$ $\displaystyle\leq
C^{\prime}|a|^{\alpha+1/2}.$

∎

###  A-B Proof Of Theorem III.2

###### Proof.

We have a wavelet base $\psi_{2}$ compactly supported and continuously
differentiable with $\int dx\psi_{2}(x)=0$. It is assumed that the
normalization of $\psi_{2}$ is $C_{\psi,\psi_{2}}=1$, the integral of the
wavelet transform is defined in Eq. (20):

$f(x)=\int_{-\infty}^{\infty}\frac{da}{a^{2}}\int_{-\infty}^{\infty}db\left\langle
f,\psi^{a,b}\right\rangle\psi_{2}^{a,b}(x).$ (20)

Then the integral would be split with two parts.

1) On the scenario of $|a|\geq 1$, the large scale item of integral is defined
in Eq. (21). The integral is uniformly bounded for $x$.

$\displaystyle\left|f_{L}(x)\right|$ $\displaystyle\leq\int_{|a|\geq
1}\frac{da}{a^{2}}\int_{-\infty}^{\infty}db\left|\psi_{2}^{a,b}(x)\right|\|f\|_{L^{2}}\|\psi\|_{L^{2}}$
(21) $\displaystyle\leq C\int_{|a|\geq
1}\frac{da}{a^{2}}\int_{-\infty}^{\infty}db|a|^{-1/2}\left|\psi_{2}\left(\frac{x-b}{a}\right)\right|$
$\displaystyle\leq
C\left\|\psi_{2}\right\|_{L^{1}}\int_{-\infty}da|a|^{-3/2}=C^{\prime}<\infty.$

We have $\left|f_{L}(x+\delta)-f_{L}(x)\right|$ for the perturbation
$|\delta|\ll 1$, the inequation is defined in Eq. (22):

$\displaystyle\left|f_{L}(x+\delta)-f_{L}(x)\right|\leq$
$\displaystyle\int_{|a|\geq
1}\frac{da}{|a|^{3}}\int_{-\infty}^{\infty}db\int_{-\infty}^{\infty}dy|f(y)|\cdot$
(22) $\displaystyle\left|\psi\left(\frac{y-b}{a}\right)\right|\cdot$
$\displaystyle\left|\psi_{2}\left(\frac{x+\delta-b}{a}\right)-\psi_{2}\left(\frac{x-b}{a}\right)\right|.$

There exists the wavelet transform on the signal
$\left|\psi_{2}(z+t)-\psi_{2}(z)\right|\leq C|t|$. For the compact support
character in the real number space $\psi_{2}\subset[-R,R],\ R<\infty$, Eq.
(23) can be deduced as follows:

$\displaystyle\left|f_{L}(x+\delta)-f_{L}(x)\right|\leq
C^{\prime}|\delta|\int_{|a|\geq
1}daa^{-4}\int_{x-b|\leq\operatorname{l}|R+1\atop|\bm{v}-b|\leq|a|R}dy|f(y)|$
(23) $\displaystyle\leq\left.C^{\prime\prime}|\delta|\int_{|a|\geq
1}da|a|^{-3}\int_{\mid y-x}\right|_{\leq 2|a|R+1}dy\quad|f(y)|$
$\displaystyle\leq C^{\prime\prime}|\delta|\|f\|_{L^{2}}\int_{|a|\geq
1}da|a|^{-3}(4|a|R+2)^{1/2}\leq C^{\prime\prime\prime}|\delta|.$

This holds for all $|\delta|\ll 1$ with the bound defined in Eq. (22).
Therefore, the function is uniform on the regular property
$\left|f_{L}(x+\delta)-f_{L}(x)\right|\leq C|\delta|$.

2) On the scenario of $|a|\leq 1$, the small scale item of integral is
satisfied with Eq. (24):

$\displaystyle\left|f_{S}(x)\right|$ $\displaystyle\leq C\int_{|a|\leq
1}\frac{da}{a^{2}}\int_{-\infty}^{\infty}db|a|^{\alpha+1/2}|a|^{-1/2}\left|\psi_{2}\left(\frac{x-b}{a}\right)\right|$
(24) $\displaystyle\leq
C\left\|\psi_{2}\right\|_{L^{1}}\int_{|a|<1}da|a|^{-1+\alpha}=C^{\prime}<\infty.$

For the perturbation $|\delta|\ll 1$, the Lipschitz regularity constraint
still holds in Eq. (25) when using again the inequation
$\left|\psi_{2}(z+t)-\psi_{2}(z)\right|\leq C|t|$.

$\begin{array}[]{l}\begin{aligned} &\left|f_{S}(x+\delta)-f_{S}(x)\right|&\\\
&\leq\int_{|a|\leq|\delta|}\frac{da}{a^{2}}\int_{-\infty}^{\infty}db|a|^{\alpha}\left(\left|\psi_{2}\left(\frac{x-b}{a}\right)\right|+\left|\psi_{2}\left(\frac{x+\delta-b}{a}\right)\right|\right)\\\
&+\int_{|\delta|\leq|a|\leq
1}\frac{da}{a^{2}}\int_{|x-b|\leq|a|R+|\delta|}db|a|^{\alpha}C\left|\frac{\delta}{a}\right|.\end{aligned}\end{array}$
(25)

The first part of integral can be written as
$C^{\prime}\left[\left\|\psi_{2}\right\|_{L^{2}}\int_{|a|\leq|\delta|}da|a|^{-1+\alpha}\right]$
while the second part of integral can be written as
$C^{\prime}\left[|\delta|\int_{|\delta|\leq|a|\leq
1}da|a|^{-3+\alpha}(|a|R+|\delta|)\right]$. The property is hold for
$\left|f_{S}(x+\delta)-f_{S}(x)\right|\leq C^{\prime\prime}|\delta|^{\alpha}$.
Then $f$ is Hölder with the exponent $\alpha$. The proof has been done.

∎

###  A-C Proof Of Theorem III.3

###### Proof.

On the assumption of $x_{0}=0$, since $\int dx\psi(x)=0$, there exists an
inequation defined in Eq. (26) which proves the theorem:

$\displaystyle\left|\left\langle f,\psi^{a,b}\right\rangle\right|$
$\displaystyle\leq\int
dx|f(x)-f(0)||a|^{-1/2}\left|\psi\left(\frac{x-b}{a}\right)\right|$ (26)
$\displaystyle\leq C\int
dx|x|^{\alpha}|a|^{-1/2}\left|\psi\left(\frac{x-b}{a}\right)\right|$
$\displaystyle\leq C|a|^{\alpha+1/2}\int
dy\left|y+\frac{b}{a}\right|^{\alpha}|\psi(y)|$ $\displaystyle\leq
C^{\prime}|a|^{1/2}\left(|a|^{\alpha}+|b|^{\alpha}\right).$

∎

###  A-D Proof Of Theorem III.4

###### Proof.

The proof is very similar with the Proof of Theorem III.2. For simplicity, we
can analyze the scenario with $|a|\leq 1$ of
$\left|f_{S}\left(x_{0}+\delta\right)-f_{S}\left(x_{0}\right)\right|$ for
small $\delta$ on the assumption that $x_{0}=0$ with the translation.

$\displaystyle\begin{array}[]{l}\left|f_{S}(\delta)-f_{S}(0)\right|\\\
\quad\leq\int_{|a|\leq|\delta|^{\alpha/\gamma}}\frac{da}{a^{2}}\int_{-\infty}^{\infty}db|a|^{\gamma}\left|\psi_{2}\left(\frac{\delta-b}{a}\right)\right|\\\
\quad+\int_{|\delta|^{\alpha/\gamma\leq|a|\leq|\delta|}}\frac{da}{a^{2}}\int_{-\infty}^{\infty}db\left(|a|^{\alpha}+\frac{|b|^{\alpha}}{|\log|b||}\right)\left|\psi_{2}\left(\frac{\delta-b}{a}\right)\right|\end{array}$
(27)
$\displaystyle+\int_{|a|\leq|\delta|}\frac{da}{a^{2}}\int_{-\infty}^{\infty}db\left(|a|^{\alpha}+\frac{|b|^{\alpha}}{|\log|b||}\right)\left|\psi_{2}\left(-\frac{b}{a}\right)\right|$
$\displaystyle+\int_{|\delta|\leq|a|}\frac{da}{a^{2}}\int_{-\infty}^{\infty}db\left(|a|^{\alpha}+\frac{|b|^{\alpha}}{|\log|b||}\right)\left|\psi_{2}\left(\frac{\delta-b}{a}\right)-\psi_{2}\left(-\frac{b}{a}\right)\right|$
.

The right hands of Eq. (27) can be devided into four parts: $T_{1}$, $T_{2}$
with support of $\psi_{2}\subset[-R,R]$, $T_{3}$, and $T_{4}$. Four separated
inequations can be derived.

$\displaystyle
T_{1}\leq\int_{|a|\leq|\delta|^{\alpha/\gamma}}da|a|^{-1+\gamma}\left\|\psi_{2}\right\|_{L^{1}}\leq
C|\delta|^{\alpha}.$ (28) $\displaystyle T_{2}$
$\displaystyle\leq\int_{|a|\leq|\delta|}da|a|^{-1+\alpha}\left\|\psi_{2}\right\|_{L^{1}}$
(29)
$\displaystyle+\int_{|\delta|^{\alpha/\gamma}\leq|a|\leq|\delta|}da|a|^{-1}\left\|\psi_{2}\right\|_{L^{1}}\frac{(|a|R+|\delta|)^{\alpha}}{|\log(|a|R+|\delta|)|}$
$\displaystyle\leq
C|\delta|^{\alpha}\left[1+\frac{1}{|\log|\delta||}\int_{|\delta|\alpha/\gamma\leq|a|\leq|\delta|}da|a|^{-1}\right]$
$\displaystyle\leq C^{\prime}|\delta|^{\alpha}.$ $\displaystyle T_{3}$
$\displaystyle\leq\int_{|a|\leq\delta}da|a|^{-1+\alpha}\left\|\psi_{2}\right\|_{L^{1}}$
(30)
$\displaystyle+\int_{|a|\leq|\delta|}da|a|^{-1}\left\|\psi_{2}\right\|_{L^{2}}\frac{(|a|R)^{\alpha}}{|\log|a|R|}$
$\displaystyle\leq C|\delta|^{\alpha}.$ $\displaystyle T_{4}$
$\displaystyle\leq C|\delta|\int_{|\delta|\leq|a|\leq 1}da|a|^{-3}$ (31)
$\displaystyle\left[|a|^{\alpha}+\frac{(|a|R+|\delta|)^{\alpha}}{|\log(|a|R+|\delta|)|}\right](|a|R+|\delta|)$
$\displaystyle\leq
C^{\prime}|\delta|\left[1+|\delta|^{-1+\alpha}+|\delta|\left(1+|\delta|^{-2+\alpha}\right)\right]\leq
C^{\prime\prime}|\delta|^{\alpha}.$

It proves the lower-order local regularity which gives an illustration of the
“mathematical microscope” of the wavelet.

∎

###  A-E Proof Of Theorem III.5

###### Proof.

There are two scenarios: $(\ell-1)2^{-j}\leq x\leq x+\delta\leq\ell 2^{-j}$,
$(\ell-1)2^{-j}\leq x\leq\ell 2^{-j}\leq x+\delta\leq(\ell+1)2^{-j}$. The
proof on the second scenario can be derived from the similar proof on the
first scenario. We then have an inequality defined in Eq. (32):

$\displaystyle|f(x)-f(x+\delta)|\leq\left|f(x)-f_{j}(x)\right|+\left|f_{j}(x)-f_{j}\left(\ell
2^{-j}\right)\right|$ (32) $\displaystyle+\left|f_{j}\left(\ell
2^{-j}\right)-f_{j}(x+\delta)\right|+\left|f_{j}(x+\delta)-f(x+\delta)\right|$
$\displaystyle\leq 2C2^{-\alpha j}+\left|f_{j}(x)-f_{j}\left(\ell
2^{-j}\right)\right|+\left|f_{j}(x+\delta)-f_{j}\left(\ell
2^{-j}\right)\right|.$

Different choices of $l$ denote different resolutions. It is assumed that
there exists a variable $k\in\mathbb{N}$ so that $x^{\prime}=x-k$ and
$\ell^{\prime}2^{-j}=\ell 2^{-j}-k$ are both in the range of $[0,1]$. If
$x^{\prime}$ is dyadic, the binary expansion of Eq. (32) is defined in Eq.
(33).

$\displaystyle\left|f_{j}(x)-f_{j}\left(\ell
2^{-j}\right)\right|\leq\left\|v_{j}\left(x^{\prime}\right)-v_{j}\left(\ell^{\prime}2^{-j}\right)\right\|$
(33) $\displaystyle=\left\|T_{d_{1}\left(x^{\prime}\right)}\cdots
T_{d_{j}\left(x^{\prime}\right)}\left[v_{0}\left(\tau^{j}x^{\prime}\right)-v_{0}\left(\tau^{j}\left(\ell^{\prime}2^{-j}\right)\right)\right]\right\|$
$\displaystyle\leq C2^{-\alpha j},$

where $\left\|\left.T_{d_{1}}\cdots T_{d_{m}}\right|_{E_{2}}\right\|\leq
C2^{-\alpha j}$ and $v_{0}$ is bounded with the property
$v_{0}(u)-v_{0}\left(u^{\prime}\right)\in E_{1}$ for all $u,u^{\prime}$. The
similar bound $\left|f_{j}(x+\delta)-f_{j}\left(\ell 2^{-j}\right)\right|$ can
also be derived. The inequation defined in Eq. (9) can be proven which is
Hölder continuous with the exponent $\alpha$.

∎