# Smooth-Reduce: Leveraging Patches for Improved Certified Robustness

Ameya Joshi Work partially done during an internship at Bosch Center for AI
(BCAI), Pittsburgh, PA. The project page can be found at https://nyu-dice-
lab.github.io/SmoothReduce/. New York University  Minh Pham  New York
University  Minsu Cho  New York University  Leonid Boytsov  Bosch Center for
AI  Filipe Condessa  Bosch Center for AI  J. Zico Kolter  Bosch Center for AI
Chinmay Hegde  New York University

###### Abstract

Randomized smoothing (RS) has been shown to be a fast, scalable technique for
certifying the robustness of deep neural network classifiers. However, methods
based on RS require augmenting data with large amounts of noise, which leads
to significant drops in accuracy. We propose a training-free, modified
smoothing approach, Smooth-Reduce, that leverages patching and aggregation to
provide improved classifier certificates. Our algorithm classifies overlapping
patches extracted from an input image, and aggregates the predicted logits to
certify a larger radius around the input. We study two aggregation schemes —
max and mean — and show that both approaches provide better certificates in
terms of certified accuracy, average certified radii and abstention rates as
compared to concurrent approaches. We also provide theoretical guarantees for
such certificates, and empirically show significant improvements over other
randomized smoothing methods that require expensive retraining. Further, we
extend our approach to videos and provide meaningful certificates for video
classifiers.

_K_ eywords Adversarial defenses, Certifiable defenses, Randomized Smoothing,
Ensemble Models, Robust Video Classifiers

##  1 Introduction

Motivation. Deep networks have been shown to be notoriously prone to “
attacks” if an adversary were allowed to modify their input [9, 26, 3]. While
several heuristic “defenses” for such attacks have been proposed [19, 41, 31],
only a handful of them are _provably accurate_ [35, 6, 22], i.e., they provide
guarantees for robust performance. The general approach in such defenses is to
_certify_ that a deep classifier, for any input data point in a volume
(parameterized by a radius) around a given input ${\mathbf{x}}$, does not
change its predictions.

In this line of work, Wong and Kolter [35] pioneered the use of bound
propagation to derive upper bounds on the certification radius for networks
with ReLU activations; however, this approach fails to scale to larger
networks, and the bounds become vacuous quite quickly. Subsequently, Cohen
_et al._ [6] and Salman  _et al._ [22] employed _randomized smoothing_ (RS) to
establish bounds on the (local) Lipschitz constant of a smoothed deep
classifier. Such approaches have since been fruitfully developed to provide
non-vacuous certificates.

Randomized smoothing (RS) methods typically involve _convolving_ the deep
classifier under consideration, $f$, with any smooth, continuous probability
distribution, ${\mathcal{P}}$ and deriving a radius of certification $R$ for
all points around ${\mathbf{x}}$ measured in some norm. For simplicity,
consider certifying models in terms of $\ell_{2}$-bounded input perturbations.
Then, a randomized smoothing scheme produces a (bound on) a certificate
parameter $R$ such that

${\mathbb{P}}\left({\mathcal{P}}\star f({\mathbf{x}})\neq{\mathcal{P}}\star
f({\mathbf{x}}+\delta)\right)\approx 0~{}~{}\text{for
any}~{}~{}\|\delta\|_{2}\leq R.$

In practice, this type of functional convolution is achieved by randomly
sampling noise vectors ${\mathbf{z}}_{i}\sim{\mathcal{P}}$, adding them to
copies of the input ${\mathbf{x}}$, and performing inference over each copy.
The resultant smooth classifier estimates the empirical probability mass,
$p_{A}$ for the correct class, $A$. Yang  _et al._ [37] derive the radius of
certification using

$R=\int_{1-p_{A}}^{1/2}\frac{1}{\Phi(p_{A})}dp_{A},$

where $p_{A}$ is the probability of the correct class under the noisy
inference, and $\Phi(\cdot)$ is the appropriate CDF. Here, the geometry of the
$\ell_{p}$ ball influences the choice of the noise distribution. For example,
Gaussian noise provides $\ell_{2}$ certificates.

|
---|---
(a) RS [22] vs. Smooth-Mean | (b) MACER [40] vs. Smooth-Mean
Figure 1: Smooth-Reduce improves upon Randomized Smoothing(RS). Smooth-Reduce
leverages patching to emulate ensembles to reduce variance of smooth
predictions by the base classifier. We study two flavors of Smooth-Reduce that
use _max_ and _mean_ aggregation schemes respectively. Smooth-Reduce takes any
base classifier that is trained to be robust to noise and uses a RS-inspired
certification algorithm to generate larger certificates with lower variants.
Our approach shows significant improvements over concurrent smoothing methods
in certified accuracy and abstention rates across several datasets and
classifiers.

The RS approach allows us to get non-trivial certificates for high-dimensional
inputs, providing the first known family of theoretically provable defenses of
deep neural network classifiers to adversarial attacks. Nevertheless, there
still remain several real-world shortcomings. First, in order for the
empirical probability mass to be accurately estimated, a large number of
samples are necessary. Second, the certified accuracy achieved via randomized
smoothing (RS) is substantially lower than empirical accuracy achieved via
heuristics such as adversarial training. Third, the addition of noise to the
inputs often significantly degrades the performance of the network. The last
problem is particularly challenging, since it requires some care to handle.
Typical RS methods (such as Salman  _et al._ [22] and Cohen  _et al._ [6])
propose noise-augmented training strategy to sidestep this problem. However,
in practice we see that noise-augmented training comes with a price: the
certified accuracies drop off significantly as the radius increases. Several
other approaches [40, 2, 11] propose improvements to prevent such a dramatic
drop-off, but they involve careful model re-training with noise augmentation,
often involving several heuristic parameters.

How then can we get better certificates? As a starting point, observe that any
RS scheme involves two basic components: the base classifier $f$ and the noise
distribution ${\mathcal{P}}$. Adding noise provides certified robustness, but
decreases accuracy; resolving this tradeoff is the key. Works such as [2, 25,
37] focus on ${\mathcal{P}}$, and propose convolution with more sophisticated
(sometimes even data-dependent) noise distributions. On the flip side, works
such as [22, 40, 1, 11, 32, 13] focus on training better base classifiers $f$.
We pursue the latter approach in this paper.

At the heart of our approach is a simple technique that is ubiquitous in
machine learning inference: _ensembling_. Aggregating results from an ensemble
of _diverse_ classifiers acting on a given data point has long been used to
improve classifier performance in standard (non-adversarial) inference
settings. However, in practice, training large (and diverse) ensembles for
deep networks can be non-trivial (and sometimes even prohibitively expensive).
The difficulty compounds when noise augmentation and adversarial training are
considered.

We overcome this difficulty by _emulating_ an ensemble classifier by
extracting a set of (large) patches from a given image, running a (single)
base classifier on all these patches, and aggregating the results. This
technique has also been successfully employed by a recent series of patch-
level models [8, 29]. Specifically, we posit that small affine transformations
of an image induce sufficient diversity leading to more robust performance.
Further, we also study two popular aggregation schemes for ensembling — _max_
and _mean_ aggregation — and demonstrate that both significantly outperform
all existing RS approaches.

Contributions. In this paper, we propose an adaptive, ensembling-based
_training-free_ smoothed classifier that significantly outperforms existing RS
methods.

Figure 2: Smooth-Reduce Certification.Smooth-Reduce modifies the RS
certification in two ways. First, an input set is created to simulate an
ensemble. In this case, we use patches sampled from the resized image.
Following the CERTIFY subroutine from [6], noise is added to every element in
the set. Next, the counts of predicted classes are aggregated to estimate
$\underline{p_{A}}$, the probability of the most probable class, $c_{A}$. The
final step uses Eq. 4 with $\underline{p_{A}}$ to derive a certificate that
holds with high probability.

Our specific contributions are as follows:

  1. 1.

We present a modified smooth classifier that leverages an input set
constructed by extracting patches of the input image, and achieves a higher
certified radius using aggregation.

  2. 2.

We show that our certificates hold with high probability with intuitive
extensions of the theoretical analysis by Cohen  _et al._ [6] and Salman _et
al._ [22].

  3. 3.

We demonstrate significant improvements in certification performance for
CIFAR-10 and ImageNet compared with several state-of-the-art randomized
smoothing approaches.

  4. 4.

Finally, we extend our approach to provide certificates for video classifiers
on UCF-101, therefore demonstrating that our approach scales to high
dimensional domains.

Techniques. Our approach consists of four steps: (1) We emulate a diverse set
of inputs from a given (single) image or video input. For our image
experiments, we simply sample overlapping contiguous patches. For our video
experiments, we sample overlapping subvideos from the original video stream.
(2) We then follow the standard randomized smoothing approach by creating $n$
copies of the input set, and adding independent noise vectors to each element
in all copies (3) We then estimate the predicted probability of each class for
each copy of the input set, and take the average of the maximum estimated
predicted probability for each copy. (4) Finally, we record certificates for
the input using the expression in Corollary 1 below. We explain each of these
steps in detail in Section 3.

##  2 Related Work

Certified Defenses. Ever since deep classifiers have been found to be
vulnerable to adversarial attacks [26, 3, 5], considerable efforts have been
directed towards developing reliable defenses [19, 41, 33, 23, 39]. The above
approaches lack strong theoretical guarantees. Provable defenses (that provide
certificates of correctness) fall into two major categories. The first
category involves establishing upper bounds on the perturbation radii for the
inputs of each layer (using linear, quadratic, convex, or even mixed-integer
programming) and propagating these bounds to achieve a certificate for an
entire network. These include works such as [35, 20, 21, 28, 14, 15, 4, 12,
34]. However, such approaches are computationally very expensive and do not
scale at all to large, modern deep network classifiers.

Randomized Smoothing. The second category of provable defense involves some
variation of randomized smoothing (RS), which advocate “smoothing” the outputs
of non-linear, non-Lipschitz networks by their functional convolution with
specially-chosen noise distributions. Early works such as [16, 6, 22] provide
$\ell_{2}$ robustness certificates by adding Gaussian noise. Subseqent works
[27, 17] have presented certificates for the $\ell_{1}$ and Wasserstein
metrics respectively using Laplacian smoothing. Yang  _et al._[37] provide a
general approach to selecting distributions for various classes of adversarial
attacks; unfortunately, certificates other than the $\ell_{2}$-norm have
$\Omega(d^{-1/2})$ dependence, leading to trivial certificates for high
dimensional inputs.

On the practical side, the above RS methods still fall short of heuristic
empirical defense methods when evaluated in terms of robust accuracy.
Therefore, several works have propose modifications to the certification
scheme to improve performance. MACER [40] maximizes surrogates of the
certified radius to train better certifiable models, while [2] finetune the
variance of noise for each input data point. [13] uses an adversarial version
of MixUp [42] to train models with better tradeoffs on accuracy and
certifiable robustness. Notice, however, that all these approaches involve re-
training large-scale models with different objectives and data augmentation
schemes.

Ensembled Defenses. Ensembling is one of the primary motivations for our
Smooth-Reduce method. We discuss some recent relevant work in that context.
Horvath  _et al._ [11] propose ensembling over diverse classifiers and show
that this decreases variance of predictions, allowing better certificates.
Yang  _et al._ [38] prove that diversified gradients and large confidence
margins are necessary and sufficient conditions for robust ensembles. Liu  _et
al._ [18] propose a weighted ensemble of networks as the base classifier and
demonstrate they provide better certificates. While all these approaches rely
on model-level ensembling, we emulate ensembles by using patching and basic
linear operations. This allows us to ensure diversity as well as improved base
classifier performance. We also demonstrate that our approach outperforms
ensemble smoothing by a large margin.

##  3 The Smooth-Reduce Framework

Preliminaries:  Let ${\mathbf{x}}\in\mathbb{R}^{d}$ be a given input. For ease
of exposition, we suppose that ${\mathbf{x}}$ is an image (and extend the
framework to video inputs later below). Let $f:\mathbb{R}^{d}\to[0,1]^{c}$ be
any classifier that takes the input and assigns each class label $c$ with
probability $f_{c}$. Cohen  _et al._ [6] propose performing inference using
the “smooth” classifier:

$\hat{f}=\operatorname*{arg\,max}_{c}\mathbb{E}_{{\mathbf{z}}\sim{\mathcal{N}}(0,\sigma^{2}{\mathbf{I}})}\left[f_{c}({\mathbf{x}}+{\mathbf{z}})\right],$
(1)

which enjoys the benefits of guarantees of correctness. To calculate these
guarantees, the standard certification approach estimates the (most probable)
class $c_{A}\in[C]$ and the second most probable class $c_{B}\in[C]$, as
predicted by $\hat{f}$. It also estimates upper and lower bounds
(respectively), $\underline{p_{A}},\overline{p_{B}}$, on the corresponding
class probabilities. To do so, we create $n_{0}$ copies of the input, add
$n_{0}$ i.i.d. Gaussian noise vectors sampled from
${\mathcal{N}}(0,\sigma^{2})$ and estimate $p_{A}$. The certified radius is
then derived using the relation:

$R=\frac{\sigma}{2}\left(\Phi^{-1}(\underline{p_{A}})-\Phi^{-1}(\overline{p_{B}})\right),$
(2)

where $\Phi^{-1}(\cdot)$ is the inverse Gaussian CDF (see [6, 22] for a
rigorous derivation). Notice that the above procedure makes no assumptions on
the base classifier $f$, and can be used to achieve a certified radius for any
model (including deep neural network classifiers).

According to Eq. 2, in order to obtain a higher radius of certification, we
can increase either the variance of the noise or the estimated probability of
the true class. However, adding large amounts of noise to the input leads to
degradation in the performance of $\hat{f}$ (compared to $f$), and could give
poor classification performance. Indeed, the majority of works focus on
training deep classifiers $f$ that are robust to noisy inputs. Instead of
pursuing this path, we focus on obtaining an improved estimate of
$\underline{p_{A}}$.

###  3.1 Smooth-Reduce

A general approach to obtaining high-quality predictions is via _ensembling_.
Using an ensemble of classifiers tends to decrease the variance of the
predicted probabilities while improving accuracy [10, P. 256]. However, deep
networks are very expensive to train, and training a large (and diverse) set
of deep classifiers for a given training dataset can be prohibitive. This
challenge is exacerbated in RS approaches which tend to require re-training
models with noise augmentation.

Instead, we draw inspiration from the folklore practice of using _cropping_
during inference to improve performance, as well as recent empirical
observations regarding the considerable effectiveness of patch-based
classification [8, 29]. We propose patching as a mechanism to create a diverse
set of (sub)images from a single input image; this allows us to emulate an
ensemble while using a single base classifier.

Our method works as follows. We create a set of inputs,
${\mathcal{X}}=\\{{\mathbf{x}}_{1},{\mathbf{x}}_{2}\dots\mathbf{x}_{k}\\}$
from a given input (base) image, ${\mathbf{x}}$, by using sampling (uniformly
at random) sub-images with $d^{\prime}$ total pixels (with $d^{\prime}<d$) and
upsampling each sub-image to the original resolution (with $d$ pixels). All
base images and patches are assumed to be square for simplicity. We then
define a modified version of the smooth classifier from Eq. 1 that we call the
“Smooth-Reduce” classifier:

$\displaystyle\bar{f}({\mathbf{x}})$
$\displaystyle=\operatorname*{arg\,max}_{c}\mathbb{E}_{{\mathbf{z}}\sim{\mathcal{N}}(0,\sigma^{2}{\mathbf{I}})}~{}\textsc{Aggregate}_{i=1}^{k}(f_{c}({\mathbf{x}}_{i}+{\mathbf{z}}))\,,$
(3)

where $\textsc{Aggregate}_{i=1}^{k}$ is a routine that _reduces_ (combines)
the predicted logits for inputs enumerated over the set ${\mathcal{X}}$. For
our approach, we consider two specific aggregation functions, _max_ and _mean_
over the predicted logits.

The Smooth-Reduce classifier, $\bar{f}$, is a simple modification of the
standard RS approach. However, we find that it improves over $\hat{f}$ in two
important aspects. Firstly, since it emulates an ensemble of classifiers, the
variance of the estimated probability $\underline{p_{A}}$ is reduced, leading
to sharper bounds on $\underline{p_{A}}$. (For a more in-depth discussion, see
also the Appendix and [11]) Further, we find that it also increases the
estimated probability values $p_{A}$ themselves; both aggregation options in
Smooth-Reduce lead to more confident classification probabilities than the
base classifier $f$. For this to hold, we have to ensure that the patches are
large enough (so that meaningful classification is achieved), and that we
extract sufficiently many patches from the input image (so that we get boosts
via aggregation).

###  3.2 Theoretical Analysis

To derive certificates of performance for our proposed Smooth-Reduce
classifier $\bar{f}$, we need to rethink Eq. 2 when used with the new class
probabilities. We first restate:

######  Theorem 1 (taken from [6]).

Let $c_{A},c_{B}\in[C]$ be the most likely and second-most likely classes, and
$\underline{p_{A}},\overline{p_{B}}\in[0,1]$ be the probability estimates
associated with $c_{A}$ and $c_{B}$. If

${\mathbb{P}}_{{\mathbf{z}}}\left(f({\mathbf{x}}+{\mathbf{z}})=c_{A}\right)\geq\underline{p}_{A}\geq\bar{p_{B}}\geq\max_{c\neq
c_{A}}{\mathbb{P}}_{{\mathbf{z}}}\left(f({\mathbf{x}}+{\mathbf{z}})=c\right),$

then $\hat{f}({\mathbf{x}}+\delta)=c_{A}$ for all vectors $\delta$ satisfying
$\|\delta\|_{2}\leq R$, where

$R=\frac{\sigma_{{\mathbf{z}}}}{2}\left(\Phi^{-1}(\underline{p}_{A})-\Phi^{-1}(\bar{p}_{B})\right).$

The above derivation for the certified radius for $\hat{f}$ does not assume
anything about the base classifier. We can therefore plug in our modified base
classifier, $\textsc{Aggregate}_{i=1}^{k}f({\mathbf{x}}_{i})$ and similarly
prove that the radius will hold with high probability.

######  Corollary 1 (Smooth-Reduce certificates).

Let $c_{A}\in[C]$, and
$\underline{p_{A}}^{\prime},\overline{p_{B}}^{\prime}\in[0,1]$ be the
probability estimates from the Smooth-Reduce classifier, $\bar{f}$. If

${\mathbb{P}}_{{\mathbf{z}}}\left(\bar{f}({\mathbf{x}}+{\mathbf{z}})=c_{A}\right)\geq\underline{p_{A}}^{\prime}\geq\overline{p_{B}}^{\prime}\geq\max_{c\neq
c_{A}}{\mathbb{P}}_{{\mathbf{z}}}\left(\bar{f}({\mathbf{x}}+{\mathbf{z}})=c\right)\,,$

then $\bar{f}({\mathbf{x}}+\delta)=c_{A}$ for all $\delta$ satisfying
$\|\delta\|_{2}\leq R$, where

$R=\frac{\sigma_{{\mathbf{z}}}}{2}\left(\Phi^{-1}(\underline{p_{A}}^{\prime})-\Phi^{-1}(\overline{p_{B}}^{\prime})\right).$
(4)

Algorithm 1 Smooth-Reduce Certification Algorithm

# certify the robustness of $\bar{f}$ around $x$

function Certify($f$, $\sigma$, $x$, $n_{0}$, $n$, $\alpha$)

$\\{x_{i}\\}$ $\leftarrow$ Patchify($x$, k)

$\texttt{counts0}\leftarrow$SmoothReduceUnderNoise$(f,\\{x_{i}\\},n_{0},\sigma)$

$\hat{c}_{A}\leftarrow$ top index in counts0

$\texttt{counts}\leftarrow\textsc{SmoothReduceUnderNoise}(f,\\{x_{i}\\},n,\sigma)$

$\underline{p_{A}}\leftarrow\textsc{LowerConfBound}$($\texttt{counts}[\hat{c}_{A}]$,
$n$, $1-\alpha$)

if $\underline{p_{A}}>\frac{1}{2}$ return prediction $\hat{c}_{A}$ and radius
$\sigma\,\Phi^{-1}(\underline{p_{A}})$

else return ABSTAIN

# Sampling with Smooth-Reduce classifiers

function SmoothReduceUnderNoise$(f,\,\\{x_{i}\\},n,\sigma)$

counts $\leftarrow[0,0,...C\text{ times}]$

for $j=1:n$

$\\{z_{i,j}\\}\rightarrow$ Sample from
${\mathcal{N}}(0,\sigma^{2}{\mathbf{I}})$

$\\{\hat{y}_{i,j}\\}=\\{f(x_{i}+z_{i,j})\\}$

$\hat{y}_{j}=\textsc{Reduce}(\\{\hat{y}_{i,j}\\})$ # Reduce over patches

counts$[\operatorname*{arg\,max}_{c\in C}\hat{y}_{j}]+=1$

return counts

Following standard practice in evaluating RS algorithms, we modify the Predict
and Certify subroutines as in [6]. For the prediction step, we create $n$
copies of our input set. We then add independent noise for each element, and
take the aggregate (maximum or average) for each class over all copies. The
classifier either returns the most likely class over the aggregated logits
over the copies, or abstains if the confidence of the probability estimate is
low. We repeat the same modifications for the certification process to
estimate lower bounds for the probability of the correct class. Alg. 1 shows a
pseudo-code representation of the certification algorithm. We also show a
diagrammatic representation of our approach in Fig. 2. For the Smooth-Max
routine, we also scale the predicted logits using softmax over the classes for
each copy. Similar to [6], our classifier abstains unless the event,
$\underline{p_{A}^{\prime}}\geq 1/2$ holds with probability larger than
$1-\alpha$.

Notice that since we are estimating the lower bound on $\underline{p_{A}}$,
the robustness guarantee holds in high probability. However, we can leverage
the benefits of ensembling in each step to improve the success probability.
For example, consider that there exists an adversarial example $\delta$ for
any sub-classifier $f_{i}$ such that $\|\delta\|_{2}\leq R$. Suppose the
probability of such an event occurring can be upper bounded by $\alpha$. Then,
the probability of $\delta$ to be an adversarial example for $\bar{f}$ is at
most $\alpha/k$; see the appendix for a detailed discussion. Theoretically,
this allows us to achieve the same performance as $\hat{f}$ with $k$ times
fewer samples. However, in practice, this may lead to a high abstention rate
if the base classifier $f$ is itself not robust enough to noise.

We now show that Smooth-Max classifiers are inherently “harder” to attack than
smooth classifiers. For this, we adapt a proof technique from Salman  _et al._
[22] to get the following result.

######  Theorem 2.

Let $\bar{f}$ and $\hat{f}$ be Smooth-Max and smooth classifiers as defined
above. Let $R_{\bar{f}}$ and $R_{\hat{f}}$ be their corresponding certified
radii derived using Alg. 1. Assuming that the correct class is $c_{A}$ and the
$|{\mathcal{X}}|$ is high enough, then, $R_{\bar{f}}\geq R_{\hat{f}}$.

The proof relies on the Smooth-Max algorithm selecting the patch with highest
probability at every step. We then use the monotonicity of $\Phi^{-1}$ to
finish the argument. We provide a detailed proof in the supplement.

Smooth-Max versus Smooth-Mean. We now reflect upon the two flavors of Smooth-
Reduce. By construction, if the base classifier succeeds on patches then
Smooth-Max should intuitively perform at least as well than standard
randomized smoothing. During inference, the Smooth-Max classifier picks the
best of possible patches in the input set. Therefore, in expectation, we hope
that the Smooth-Max classifier will be more robust. The Smooth-Mean
classifier, on the other hand, improves predictions using averaging to reduce
the variance. Intuitively, patches that are classified with low confidence are
countered by patches with very high confidence. Our observation is that the
Smooth-Mean classifier abstains less frequently as compared to Smooth-Max and
base RS classifiers. This also showcases one of the limitations of Smooth-Max
classifiers: if the base classifier, $f$ is very robust to noise, then a bad
patch can consistently be chosen leading to the Smooth-Max classifier
abstaining more often. In such a case, the Smooth-Mean classifier rectifies
this by not relying on a single patch, leading to fewer abstentions. We also
see evidence of this behavior in our results as discussed below.

##  4 Experiments and Results

###  4.1 Certificates for Image Classifiers

We evaluate our approach by certifying classifiers trained on CIFAR-10 and
ImageNet. To meaasure the performance, we consider three metrics: (1) The
approximate certified accuracy with respect to the radius, (2) Average
Certified Radius (ACR), and (3) the abstention rate. We define average
certified radius as in [40], where for each $(x_{i},y_{i})$ in the test set,
$D_{\text{test}}$, and the corresponding certified radius, $R_{i}$, we
calculate the ACR as
$\frac{1}{|D_{\text{test}}|}\sum_{(x_{i},y_{i}}\mathbf{1}[\bar{f}(x_{i})=y_{i}]R_{i}$.
The abstention rate is defined as the fraction of abstentions for the given
test set. We show that our approach improves upon all the metrics over other
randomized smoothing methods.

Setup: In order to test our approach, we leverage the base classifiers trained
in [22] for CIFAR-10 and ImageNet. The base classifiers have been
adversarially trained to be robust to varying Gaussian distributions as well
as smooth adversarial attacks. We use the models with the highest reported
performance for each value of the variance, $\sigma$. Further, we use
$n_{0}=100$ samples for prediction, and $n=100,000$ for certifying CIFAR-10,
and similarly $n_{0}=100,n=100,000$ for Imagenet. We choose the best reported
models in [40, 2] and [11] for comparisons with the same setting unless
otherwise stated. We retrained models for MACER [40] for CIFAR-10 and
Imagenet. However, for others, we used reported numbers as the code and models
were not available at the time of publication. Additional details are
available in the appendix and the accompanying supplementary material.

Figure 3: Certified Accuracies for Cifar-10.Smooth-Max classifiers provide
better certificates as compared to other approaches. For the same number of
copies of the input set, we see that Smooth-Reduce outperforms both in terms
of certified radius and also abstains less frequently than all other
approaches. Additionally, Smooth-Reduce can be effortlessly integrated with
other improvements in RS (for example, MACER [40]) without any retraining to
achieve improved certificates.

Results on CIFAR-10: For CIFAR-10, we use the pretrained Resnet-110 from [22].
Since the inputs are required to be $32\times 32$ images, we resize each input
image to be $36\times 36$ and sample $25$ patches of the size $32\times 32$
using either a random or uniform sampling process. We certify the first $500$
CIFAR-10 test images with both variants of our Smooth-Reduce algorithm for
$\sigma={0.25,0.5,1.0}$. Certificates for a larger sample of the dataset are
also available in the Appendix. Note that we use the corresponding
adversarially trained noise models from [22] as our base models. Fig. 3 shows
results our experiments. We also compare Smooth-Reduce with standard
randomized smoothing [6, 22], DDRS [2], MACER [40] and Ensemble smoothing
[11]. We demonstrate that the Smooth-Mean and Smooth-Max algorithms outperform
all other approaches by a significant margin in terms of certified accuracy.
Further, the average certified radius for Smooth-Mean and Smooth-Max exceed
that of other approaches by at least 25% and 14% respectively. We also see
that the improvements increase in magnitude as the noise variance increases.
Finally, also note that, we are also successful in reducing abstention rates.
Table 1 shows the average certified radii and abstention rates for the various
certification algorithms. Also see that Smooth-Mean classifiers tend to
abstain far less often as compared to Smooth-Max classifiers.

We also study the effect of confidence calibration by ensuring both SmoothAdv
and Smooth-Reduce classifiers see the same number of overall patches. To
ensure fair comparison, we run SmoothAdv certification with $N=100k$ and
$\alpha=0.001$ and Smooth-Reduce with $N=10,000,k=10,$ and use $\alpha$ to be
$0.01$ for each sub-classifier. Fig. 5(b) shows that under the same number of
samples, Smooth-Reduce is able to certify larger radii while improving the
certified accuracy.

Table 1: Results for CIFAR-10.Average Certified Radii and Abstention rates for CIFAR-10. The best performer is bolded and the second best is italicized. Results with a $*$ are from reported results in the publication. Algorithm | ACR $\uparrow$ | Abst. Rate $\downarrow$
---|---|---
$\sigma$ | 0.25 | 0.5 | 1.0 | 0.25 | 0.5 | 1.0
Smooth-Max (Ours) | 0.713 | 1.269 | 2.04 | 0.017 | 0.047 | 0.1064
Smooth-Mean (Ours) | 0.759 | 1.426 | 2.665 | 0.005 | 0.046 | 0.030
Smooth-Max Macer (Ours) | 0.701 | 1.209 | 1.910 | 0.018 | 0.058 | 0.110
Smooth-Mean Macer (Ours) | 0.754 | 1.462 | 2.736 | 0.005 | 0.0136 | 0.020
Standard RS [6, 22] | 0.605 | 0.899 | 1.185 | 0.039 | 0.1099 | 0.2582
MACER [40] | 0.517 | 0.682 | 0.767 | 0.206 | 0.366 | 0.576
DDRS [2] | 0.678 | 0.942 | 1.185 | 0.048 | 0.122 | 0.244
Ensemble RS∗ [11] | 0.583 | 0.756 | 0.788 | - | - | -
Table 2: Results for Imagenet.Average Certified Radius and Abstention rates for ImageNet. The best performer is bolded and the second best is italicized. Results with a $*$ are reported results in the publication.  Algorithm | ACR $\uparrow$ | Abst. Rate $\downarrow$
---|---|---
$\sigma$ | 0.25 | 0.5 | 1.0 | 0.25 | 0.5 | 1.0
Smooth-Max (Ours) | 0.767 | 1.453 | 2.611 | 0.008 | 0.038 | 0.108
Smooth-Mean (Ours) | 0.786 | 1.513 | 2.931 | 0.002 | 0.024 | 0.048
Standard RS [6, 22] | 0.729 | 1.327 | 2.204 | 0.02 | 0.098 | 0.22
MACER∗ [40] | 0.544 | 0.831 | 1.008 | - | - | -
Ensemble RS∗ [11] | 0.545 | 0.868 | 1.108 | - | - | -

Results on Imagenet: We also test our approach on $500$ randomly chosen images
from Imagenet to certify a pretrained Resnet-20 model from [22]. We resize our
inputs to $256\times 256$ and sample $224\times 224$ sub-patches. We use
$4,8,$ and $16$ patches for our approach with $n_{0}=100$ and $n=100k$ for
certification. Smooth-Reduce improves upon standard randomized smoothing as
shown in Fig. 4. Specifically, Smooth-Mean performs the best, having 32.9%
relatively higher average certified radius as compared to standard randomized
smoothing. Smooth-Max performs the second-best. Also notice that Smooth-Max
performance remains stable with the number of patches.

Figure 4: Certification for ImageNet.We see similar performance improvements
of Smooth-Reduce over standard RS. Increasing number of patches does not
affect Smooth-Max certificates significantly. However, Smooth-Mean classifiers
have a higher approximate certified accuracy as the number of patches
increase, especially as the noise variance increases. More detailed results
can be found in the appendix.

###  4.2 Certificates for Video Classifiers

Video classifiers often employ aggregation over chunks from the video stream
to tackle the problem of varying number of frames [7](see Fig. 9 in appendix).
Therefore, we propose Smooth-Reduce as a natural method to certify such
classifiers. While standard RS certificates have not been reported for such
models, we observe in Fig. 6(a) and Fig. 10(see Appendix) that certified
accuracies using RS are still low. As a remedy, we adapt our Smooth-Reduce
algorithm to videos.

While the natural approach here would be to simply look at overlapping chunks
as analogues for patches, initial tests showed catastrophic loss of accuracy
when we use single chunks for prediction. We therefore sample overlapping sub-
videos with $t$ frames instead. Each sub-video consists of a fixed number of
chunks; each with $m$ frames. The base video classifier aggregates over these
chunks to produce a prediction. We repeat the same process of smoothing and
aggregation over the sub-videos instead of chunks, and label this as Smooth-
Reduce-$(t,m)$ where $t$ is the number of frames in each sub-video and $m$ is
the number of frames in each chunk. Fig. 9 in the appendix shows a pictorial
representation;see Sec. B for a more detailed description. Note here that the
base classifier itself is an ensemble over multiple $16$ frame chunks.

Experiments and Results. We test our approach on 3D ResNeXt-101 RGB [36]
trained on UCF-101 [24].We retrain models initialized with weights from [7]
using clips of 16 consecutive RGB frames with Gaussian noise augmentation.
Similar to the setting of [7], we use SGD with weight decay of $0.0005$,
momentum of $0.9$, and initial learning rate of $0.1$. We used the first train
split and the first test split for training and testing our model,
respectively. Additional training details can be found in the appendix.

For inference, the video classifier [7] follows these steps: (1) the input
video stream is split into non overlapping chunks of $16$ frames each, (2) the
model predictions on these chunks are averaged, and returned as the output
class. We run Smooth-Reduce certification by first sampling $64$ frame or
$128$ frame sub-videos for a video stream (analogous to patching for images)
to create the input set. Then we plug in the video classifier inference
routine to predict classes for noisy copies of each sub-video. See Fig. 6 for
results. Note that Smooth-Max and Smooth-Mean both outperform the standard
randomized smoothing classifier.

Limitations of Randomized Smoothing for Videos. We encountered several
challenges while attempting to certify video classifiers. A significant
challenge was training noise robust classifiers. We observe that adding
Gaussian noise to video data often led to catastrophic decreases in accuracy.
This could be an artifact of the architecture which averages predictions over
frames by itself. Further, the memory requirements often became insurmountable
to get high probability certificates. Our certificates here have been
estimated using $n_{0}=10$ samples for prediction, and $n=1000$ samples for
certification with a failure probability of $\alpha=0.001\%$. However, Smooth-
Reduce allows for lower sample complexity (see appendix), allowing for Smooth-
Mean models to still achieve non-trivial certified accuracies.

|
---|---
Figure 5: Effect of resizing.(L) Notice that as the input size increases, the
information in each patch correspondingly decreases. The base classifier
performs worse overall for each patch leading to lower certified accuracy. (b)
Certification with Confidence Calibration (R). Under the same number of
overall samples and calibrated failure probabilites, Smooth-Reduce out-
performs standard RS [22] in both certification radius and certified accuracy.

###  4.3 Ablation Studies.

We further analyse the various components of our approach.

Number of patches: We measure the effect of the number of samples used for
Smooth-Reduce certification on Imagenet classifiers. Fig. 4 shows that Smooth-
Max classifiers are relatively unaffected by the number of samples chosen.
However, we see that increasing number of patches improves performance of
Smooth-Mean certificates. This can be attributed to better empirical estimates
as the number of samples increase, and also verifies our theoretical analysis;
see appendix. Note too that this difference is more evident for higher noise
variances.

Effect of Resizing: Another component of Smooth-Reduce is the resizing step
undertaken while sampling. While theoretically it should not affect the
radius, the base classifier does assume that the features would be of a
certain size. We analyse the effect of the resizing step by resizing Imagenet
test images to $384\times 384,512\times 512$ and, $640\times 640$ and sampling
$16$ patches of $224\times 224$ randomly. In Fig. 5(a), we observe that as
resizing becomes more extreme, the certified accuracy falls in tandem with
base accuracy.

Random v/s dense sampling: Since sampling of patches plays a large role in
creating a diverse input set, we also analyse the effect of two sampling
approaches; dense and uniform random. For random sampling, we select patches
randomly with replacement from the resized input image, and discarding any
‘invalid’ patches that fall outside the image borders. For dense sampling, we
sample overlapping patches with a specified stride length. We evaluate if the
sampling approach affects the certificates by sampling $25$ patches for each
method. We observe that the sampling process does not affect the certification
process as long as the number of patches are high enough (see Fig. 7 in
appendix).

Effect of subvideo/chunk sizes: We analyze the effect using different subvideo
and chunk sizes on UCF101 certifcation. Specifically, we use either 64 or 128
frames for the size of subvideos, and 8 or 16 frames for the size of chunks.
Fig. 6 suggest that we would benefit from having a larger subvideo or chunk
sizes. In fact, we observe that using 128 frame subvideos and 64 frame chunks
yield the highest certification accuracy.

|
---|---
Figure 6: Results on UCF-101. For standard RS classifiers, the number
represents the number of frames in each chunk. For Smooth-Reduce classifiers,
the first number is the subvideo size, and the second is the number of frames
in each chunk. (Left) Varying chunk sizes.Our modified Smooth-Reduce
classifiers provide higher certified accuracies as compared to standard RS on
UCF-101 videos. Also observe that for a fixed subvideo size, RS and Smooth-
Reduce classifiers using larger chunks (16 frames over 8 frames) are more
robust. (Right) Varying subvideo sizes. Smooth-Reduce classifiers using larger
subvideo sizes are more robust. The difference is more pronounced for Smooth-
Mean classifiers which outperform standard RS by a large margin. More results
can be found in the Appendix.

##  5 Discussion and Conclusions

We present Smooth-Reduce, an extension of the randomized smoothing approach
proposed in [6]. We empirically and theoretically proved that Smooth-Reduce
classifiers improve over standard randomized smoothing in terms of certified
radii, as well as abstention rate. Our approach relies on the performance
boosting properties of ensemble classifiers, which we emulate by creating an
input set using patches. We show better certification performance as compared
to other smoothing methods under two different aggregation schemes. A major
benefit is that our approach requires no additional re-training.

Our approach also does not make any assumptions on the base classifier.
Therefore, Smooth-Reduce can plugged in effortlessly into other certified
classifiers, such as MACER [40], and achieve improved certificates. Some
limitations of our approach persist. Firstly, we require higher inference-time
computation than standard RS approaches. However, note that in comparison to
other ensembling approaches [11, 38], our method does not require training
multiple classifiers. Further, we have not studied adaptive attacks for this
scheme, and constructing reasonable attacks for such classifiers (and
verifying these certificates empirically) is a complex research question in
and of itself. We leave these directions to future work.

## Acknowledgements

Part of this work was performed when AJ was a summer intern at Bosch AI, where
it was supported by DARPA grant HR11002020006. AJ, MC, MP, and CH would also
like to acknowledge NSF grants CCF-2005804 and CCF-1815101, USDA/NIFA grant
2021-67021-35329, and ARPA-E DIFFERENTIATE grant DE-AR0001215.

## References

  * [1] Addepalli, S., Jain, S., Sriramanan, G., Babu, R.V.: Boosting adversarial robustness using feature level stochastic smoothing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 93–102 (2021)
  * [2] Alfarra, M., Bibi, A., Torr, P.H.S., Ghanem, B.: Data dependent randomized smoothing. ArXiv abs/2012.04351 (2020)
  * [3] Athalye, A., Carlini, N., Wagner, D.: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In: ICML (2018)
  * [4] Carlini, N., Katz, G., Barrett, C., Dill, D.L.: Ground-truth adversarial examples. arXiv (2017)
  * [5] Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. IEEE (SP) (2017)
  * [6] Cohen, J., Rosenfeld, E., Kolter, Z.: Certified adversarial robustness via randomized smoothing. In: ICML. PMLR (2019)
  * [7] Crasto, N., Weinzaepfel, P., Alahari, K., Schmid, C.: Mars: Motion-augmented rgb stream for action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7882–7891 (2019)
  * [8] Dosovitskiy, A., Beyer, L., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2020)
  * [9] Goodfellow, I.: Defense against the dark arts: An overview of adversarial example security research and future research directions. arxiv preprint 1806.04169 (2018)
  * [10] Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016), http://www.deeplearningbook.org
  * [11] Horv’ath, M.Z., Müller, M.N., Fischer, M., Vechev, M.T.: Boosting randomized smoothing with variance reduced classifiers. ArXiv abs/2106.06946 (2021)
  * [12] Huang, X., Kwiatkowska, M., Wang, S., Wu, M.: Safety verification of deep neural networks. Computer Aided Verification (CAV) (2017)
  * [13] Jeong, J., Park, S., Kim, M., Lee, H.C., Kim, D.G., Shin, J.: Smoothmix: Training confidence-calibrated smoothed classifiers for certified robustness. Advances in Neural Information Processing Systems 34 (2021)
  * [14] Katz, G., Barrett, C., Dill, D., Julian, K., Kochenderfer, M.: Reluplex: An efficient smt solver for verifying deep neural networks. arXiv preprint arXiv:1702.01135 (2017)
  * [15] Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Towards proving the adversarial robustness of deep neural networks. arXiv preprint (2017)
  * [16] Lecuyer, M., Atlidakis, V., Geambasu, R., Hsu, D., Jana, S.: Certified robustness to adversarial examples with differential privacy. In: 2019 IEEE Symposium on Security and Privacy (SP). pp. 656–672. IEEE (2019)
  * [17] Levine, A.J., Feizi, S.: Improved, deterministic smoothing for $L_{1}$ certified robustness. In: ICML (2021)
  * [18] Liu, C., Feng, Y., Wang, R., Dong, B.: Enhancing certified robustness of smoothed classifiers via weighted model ensembling. ArXiv abs/2005.09363 (2020)
  * [19] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: ICLR (2018), https://openreview.net/forum?id=rJzIBfZAb
  * [20] Raghunathan, A., Steinhardt, J., Liang, P.: Certified defenses against adversarial examples. In: ICLR (2018)
  * [21] Raghunathan, A., Steinhardt, J., Liang, P.S.: Semidefinite relaxations for certifying robustness to adversarial examples. In: NeurIPS (2018)
  * [22] Salman, H., Yang, G., Li, J., Zhang, P., Zhang, H., Razenshteyn, I., Bubeck, S.: Provably robust deep learning via adversarially trained smoothed classifiers. In: NeurIPS (2019)
  * [23] Samangouei, P., Kabkab, M., Chellappa, R.: Defense-gan: Protecting classifiers against adversarial attacks using generative models. ArXiv abs/1805.06605 (2018)
  * [24] Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild (2012)
  * [25] Súkeník, P., Kuvshinov, A., Günnemann, S.: Intriguing properties of input-dependent randomized smoothing. arXiv preprint arXiv:2110.05365 (2021)
  * [26] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. International Conference on Learning Representations (2014)
  * [27] Teng, J., Lee, G.H., Yuan, Y.: $\ell_{1}$ adversarial robustness certificates: a randomized smoothing approach. OpenReview (2019), https://openreview.net/forum?id=H1lQIgrFDS
  * [28] Tjeng, V., Xiao, K., Tedrake, R.: Evaluating robustness of neural networks with mixed integer programming. arXiv preprint arXiv:1711.07356 (2017)
  * [29] Trockman, A., Kolter, J.Z.: Patches are all you need? ArXiv abs/2201.09792 (2022)
  * [30] Vershynin, R.: High-dimensional probability: An introduction with applications in data science, vol. 47. Cambridge university press (2018)
  * [31] Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., Gu, Q.: Improving adversarial robustness requires revisiting misclassified examples. In: ICLR (2019)
  * [32] Wang, Y.: Improving adversarial robustness for free with snapshot ensemble. ArXiv abs/2110.03124 (2021)
  * [33] Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., Gu, Q.: Improving adversarial robustness requires revisiting misclassified examples. In: ICLR (2019)
  * [34] Weng, T.W., Zhang, H., Chen, H., Song, Z., Hsieh, C.J., Boning, D.S., Dhillon, I.S., Daniel, L.: Towards fast computation of certified robustness for relu networks. In: ICML (2018)
  * [35] Wong, E., Kolter, Z.: Provable defenses against adversarial examples via the convex outer adversarial polytope. In: ICML. PMLR (2018)
  * [36] Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks (2017)
  * [37] Yang, G., Duan, T., Hu, E.J., Salman, H., Razenshteyn, I.P., Li, J.: Randomized smoothing of all shapes and sizes. In: ICML (2020)
  * [38] Yang, Z., Li, L., Xu, X., Kailkhura, B., Xie, T., Li, B.: On the certified robustness for ensemble models and beyond. ArXiv abs/2107.10873 (2021)
  * [39] Yin, Z.and Wang, H., Wang, J., Tang, J., Wang, W.: Defense against adversarial attacks by low-level image transformations. International Journal of Intelligent Systems (2020)
  * [40] Zhai, R., Dan, C., He, D., Zhang, H., Gong, B., Ravikumar, P., Hsieh, C.J., Wang, L.: Macer: Attack-free and scalable robust training via maximizing certified radius. ArXiv abs/2001.02378 (2020)
  * [41] Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: ICML. pp. 7472–7482 (2019)
  * [42] Zhang, H., Cissé, M., Dauphin, Y., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. ArXiv abs/1710.09412 (2018)

##  Appendix A Results and Discussion

We support our observations in Sec. 4 with some additional results presented
here.

###  A.1 Additional Results for CIFAR-10

Table 3: Detailed results on CIFAR-10.

$\sigma$ | Approach | Radii |  |
---|---|---|---|---
|  | 0.0 | 0.25 | 0.5 | 0.75 | 1.0 | 1.25 | 1.5 | 1.75 | 2.0 | 2.25 | 2.5 | 2.75 | 3.0 | 3.25 | 3.5 | 3.75 | 4.0
0.25 | Smooth-Max (Ours) | 85 | 79 | 74 | 65 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Smooth-Mean (Ours) | 84 | 82 | 79 | 76 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
SmoothAdv [22] | 84 | 73 | 58 | 39 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
MACER-Smooth-Max (Ours) | 80 | 75 | 68 | 60 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
MACER-Smooth-Mean (Ours) | 80 | 78 | 75 | 71 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
MACER [40] | 79 | 67 | 52 | 40 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
DDRS [2] | 73 | 61 | 51 | 39 | 18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| Ensemble [11] | 83 | 70 | 55 | 42 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
0.50 | Smooth-Max (Ours) | 73 | 69 | 64 | 60 | 55 | 49 | 43 | 36 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Smooth-Mean (Ours) | 74 | 72 | 69 | 67 | 66 | 64 | 61 | 58 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
SmoothAdv [22] | 72 | 61 | 50 | 40 | 31 | 20 | 10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
MACER-Smooth-Max (Ours) | 67 | 62 | 59 | 54 | 48 | 41 | 35 | 26 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
MACER-Smooth-Mean (Ours) | 67 | 66 | 64 | 62 | 60 | 58 | 56 | 53 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
MACER [40] | 63 | 56 | 47 | 43 | 34 | 25 | 20 | 11 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
DDRS [2] | 61 | 53 | 45 | 35 | 27 | 19 | 13 | 7 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| Ensemble [11] | 65 | 59 | 49 | 45 | 38 | 32 | 26 | 19 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
1.0 | Smooth-Max (Ours) | 54 | 50 | 47 | 44 | 41 | 37 | 34 | 31 | 27 | 23 | 20 | 18 | 15 | 13 | 10 | 7 | 0
Smooth-Mean (Ours) | 57 | 55 | 54 | 52 | 49 | 48 | 46 | 45 | 44 | 42 | 41 | 39 | 37 | 34 | 32 | 28 | 0
SmoothAdv [22] | 50 | 44 | 36 | 28 | 21 | 18 | 13 | 9 | 7 | 5 | 4 | 2 | 1 | 0 | 0 | 0 | 0
MACER-Smooth-Max (Ours) | 45 | 42 | 39 | 35 | 33 | 30 | 27 | 23 | 22 | 19 | 16 | 13 | 11 | 9 | 7 | 3 | 0
MACER-Smooth-Mean (Ours) | 44 | 44 | 42 | 41 | 40 | 40 | 39 | 39 | 38 | 36 | 36 | 35 | 35 | 33 | 31 | 28 | 0
MACER [40] | 42 | 39 | 35 | 32 | 30 | 27 | 25 | 20 | 18 | 15 | 12 | 9 | 7 | 6 | 4 | 1 | 0
DDRS [2] | 46 | 39 | 33 | 27 | 22 | 19 | 15 | 12 | 9 | 5 | 4 | 3 | 2 | 1 | 0 | 0 | 0
| Ensemble [2] | 49 | 43 | 37 | 30 | 23 | 18 | 16 | 13 | 11 | 9 | 5 | 0 | 0 | 0 | 0 | 0 | 0

###  A.2 Additional Results for ImageNet

Table 4: Detailed results on ImageNet. $\sigma$ | Approach | Radii
---|---|---
|  | 0.0 | 0.5 | 1.0 | 1.5 | 2.0 | 2.5 | 3.0 | 3.5 | 4.0
0.25 | Smooth-Max-16(PGD) | 65 | 61 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Smooth-Mean-16 (PGD) | 64 | 63 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Smooth-Max-16 (DDN) | 72 | 67 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Smooth-Mean-16 (DDN) | 72 | 69 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Smooth-Max-8(PGD) | 65 | 61 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Smooth-Mean-8 (PGD) | 65 | 62 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Smooth-Max-8 (DDN) | 72 | 67 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Smooth-Mean-8 (DDN) | 72 | 68 | 0 | 0 | 0 | 0 | 0 | 0 | 0
SmoothAdv [22] | 61 | 55 | 0 | 0 | 0 | 0 | 0 | 0 | 0
0.50 | Smooth-Max-16(PGD) | 57 | 54 | 50 | 48 | 0 | 0 | 0 | 0 | 0
Smooth-Mean-16 (PGD) | 58 | 56 | 54 | 52 | 0 | 0 | 0 | 0 | 0
Smooth-Max-16 (DDN) | 66 | 61 | 56 | 51 | 0 | 0 | 0 | 0 | 0
Smooth-Mean-16 (DDN) | 66 | 64 | 61 | 59 | 0 | 0 | 0 | 0 | 0
Smooth-Max-8 (PGD) | 57 | 54 | 50 | 46 | 0 | 0 | 0 | 0 | 0
Smooth-Mean-8 (PGD) | 56 | 55 | 53 | 50 | 0 | 0 | 0 | 0 | 0
Smooth-Max-8 (DDN) | 66 | 61 | 56 | 48 | 0 | 0 | 0 | 0 | 0
Smooth-Mean-8 (DDN) | 65 | 63 | 59 | 54 | 0 | 0 | 0 | 0 | 0
SmoothAdv [22] | 51 | 47 | 40 | 32 | 0 | 0 | 0 | 0 | 0
1.0 | Smooth-Max-16(PGD) | 44 | 41 | 38 | 37 | 34 | 31 | 28 | 0 | 0
Smooth-Mean-16 (PGD) | 44 | 44 | 42 | 41 | 39 | 38 | 36 | 0 | 0
Smooth-Max-16 (DDN) | 54 | 50 | 46 | 42 | 37 | 33 | 28 | 0 | 0
Smooth-Mean-16 (DDN) | 55 | 53 | 51 | 50 | 47 | 43 | 38 | 0 | 0
Smooth-Max-8 (PGD) | 42 | 41 | 38 | 36 | 33 | 30 | 27 | 0 | 0
Smooth-Mean-8 (PGD) | 45 | 43 | 40 | 39 | 37 | 35 | 33 | 0 | 0
Smooth-Max-8 (DDN) | 54 | 49 | 44 | 39 | 35 | 31 | 25 | 0 | 0
Smooth-Mean-8 (DDN) | 54 | 53 | 50 | 46 | 42 | 37 | 34 | 0 | 0
SmoothAdv [22] | 37 | 33 | 29 | 26 | 23 | 18 | 16 | 0 | 0

Random Sampling versus Dense Sampling. In order to understand the effect of
sampling, we analyse the performance of Smooth-Reduce on CIFAR-10 under two
sampling schemes: (1) randomly sampling patches under an Uniform distribution,
and, (2) densely sampling patches with a specific stride length. Intuitively,
the two sampling schemes should not affect performance given enough number of
patches. Fig. 7 shows that this conjecture holds, with both Smooth-Max and
Smooth-Mean presenting comparable performance under the two sampling schemes.

Figure 7: Effect of Sampling Algorithm  The sampling algorithm does not affect
the certified accuracy in any significant manner for both Smooth-Reduce
classifiers, suggesting that the only hyperparameter of consequence is the
number of patches ($k$).

Performance with the same number of inferences. An important question that
arises is if the improved certification performance is an artifact of the
higher number of samples. We show that this is not the case by certifying
SmoothAdv [22] and Smooth-Reduce with the same number of samples, $N=100k$.
For ensuring fair comparison, we reduce the failure rate probability rate per
Smooth-Reduce sub-classifier to $\alpha=0.01$ in comparison to $\alpha=-0.001$
for SmoothAdv. As we observe in Fig. 8, we achieve higher certified accuracies
as well as better certified radii, given the same amount of compute.

Figure 8: Confidence calibrated Smooth-Reduce

##  Appendix B Video Classifiers

Video classifiers come in a large variety of flavors; 3d convolutional ,
hybrid conv-LSTM models , optical flow-based models , and others. In this
paper, we only focus on certifying pure RGB frame based models. This is both
due to the models being less computationally expensive as well as achieving
high benign performance without a large amount of heuristic tuning. We
specifically use the RGB ResNext-101 models from [7] for certifying UCF-101
videos. [7] propose a hybrid RGB-optical flow model as well, which we propose
can be adapted easily to a wide variety of video classification tasks. They
train two ResNext-101 modesl with 3D convolutions, the first on RGB frame
chunks, and, the second on optical flow representations. We just use the first
model for certification. However, randomised smoothing for such jointly
trained multi-model classifiers is a separate and interesting technical
discussion in itself.

Training. For training, we initialize our ResNext-101 with weights from the
model in [7] pretrained on the Kinetics dataset. We then train the 3D CNN with
$8$ or $16$ frame chunks from the UCF-101 training set. Following [7], we use
SGD with weight decay of $0.0005$, momentum of $0.9$, and initial learning
rate of $0.1$. In order to make the classifiers robust to Gaussian noise, we
also use Gaussian noise augmentation similar to [6]. Further, we also use the
noise-variance scheduling scheme presented in [22], by slowly incrementing
noise from $0$ to the required noise levels every $20$ epochs. For inference,
the video classifier averages the logits of non-overlapping $8$ or $16$ frame
chunks sequentially sampled from the video stream. A pictorial depiction can
be seen in Fig. 9. We used the first train split and the first test split for
training and testing our model, respectively. Our base model achieves $\sim
86\%$ benign accuracy on the testset.

For Smooth-Reduce prediction, we follow the procedure presented above in Sec.
4.2 by modifying the inference step. We first sample $k$ overlapping sub-
videos from the original test video-stream. For our experiments, we use $64$
and $128$ frame subvideos. Next, we create $n$ copies for each sub-video and
run the base video inference described above with $16$ or $8$ frame chunks for
each noisy copy. The predictions are then aggregated over the copies using the
selected Aggregate (_max_ /_mean_) Smooth-Reduce methods. The algorithm then
returns the class with the largest count. We show results of this in Fig. 10
for noise varinaces of $0.25,0.5,$ and $1.0$. Notice that while the certified
radii are still somewhat low, Smooth-Reduce outperforms standard Randomized
smoothing, certifying not only larger radii but also providing greater
certified accuracies. We also see that higher chunk sizes and sub-video sizes
result in better certification performance in terms of certified accuracy.

Figure 9: Smooth-Reduce for Videos: Video classifiers include averaging over
frames or chunks of frames. Observing that larger chunk sizes provide better
certificates, Smooth-Reduce takes this a step further by first sampling
overlapping sub-videos with $4$ or $8$ chunks of $16$ frames each. We then
aggregate the smooth predictions over sub-videos.
---
Figure 10: Additional video results. Notice that Smooth-Mean certifies larger
radii while presenting higher certified accuracies. Another point of interest
is that larger sub-videos and larger chunks show better certification
performance. Model nomenclature is as follows; for standard randomized
smoothing, models are named as _Standard RS_ -chunk-size; for Smooth-Reduce,
we use _Smooth-{max/mean}_-sub-video size, chunk size, in terms of number of
frames.

##  Appendix C Deferred Theorems and Proofs

######  Theorem 3 (Smooth-Max classifiers certify larger radii).

Let $\hat{f}$ and $\bar{f}$ be the standard RS classifier and the Smooth-Max
classifier as defined, if $R_{\hat{f}}$ and $R_{\bar{f}}$ represent the
certified radii for the two classifiers for a specific input, ${\mathbf{x}}$,
then

$\displaystyle R_{\bar{f}}$ $\displaystyle\geq R_{\hat{f}}$

for all ${\mathbf{x}}$.

###### Proof.

Assume that the correct class predicted by both $\hat{f}$ and $\bar{f}$ is
$A$. Let $\underline{p_{A}}$ be the probability estimate for the $\hat{f}$
smooth classifier, and $\underline{p_{A}^{\prime}}$ be that for $\bar{f}_{4}$
for $n$ samples.

We make a minor assumption here, that the input set is large enough that it
contains the original image, ${\mathbf{x}}_{c}={\mathbf{x}}$. This can be
easily enforced by appending the original image to the input set.

Now,

$\displaystyle\underline{p_{A}}$
$\displaystyle=\frac{1}{n}\sum_{j=1}^{n}f({\mathbf{x}}_{c}+{\mathbf{z}}_{j})$
(5) $\displaystyle\underline{p_{A}^{\prime}}$
$\displaystyle=\frac{1}{n}\sum_{j=1}^{n}\max_{i}f({\mathbf{x}}_{i}+{\mathbf{z}}_{j})$
(6)

, where ${\mathbf{x}}_{c}$ refers to the center-crop of the resized
${\mathbf{x}}$.

By definition, $\underline{p_{A}^{\prime}}$ will always be greater than or
equal to $\underline{p_{A}}$. Therefore, the above statement holds true. ∎

We also prove that Smooth-Reduce classifiers have a lower failure probability
for a given perturbation $\delta$.

######  Theorem 4 (Smooth-Reduce confidence bounds).

Let $\hat{f}$ and $\bar{f}$ be the smooth and Smooth-Reduce classifiers
defined above. Let $f_{i}$ be the sub-classifiers in $\bar{f}$. Let $R$ be the
certified radius for $\bar{f}$ w derived using the Smooth-Reduce Certify
subroutine with $n$ samples and $k$ patches, with probability $\alpha_{1}$.
Let $R_{i},~{}i=1:k$ be the same for the sub-classifiers, $f_{i}$, derived
using standard smoothing certification with $n$ samples with probability
$\alpha$. Then, for Smooth-Mean classifiers, $\alpha_{1}\leq
e^{-k\alpha}(2e\alpha)^{k/2}$

###### Proof.

Assume that our Smooth-Mean classifer, $\bar{f}$ CERTIFY method returns some
certified radius, $R$ with the correct class, $A$ for the given number of
samples, $N$ and patches, $p$. Further, we can use CERTIFY from [6] to
estimate certified radii, $R_{i}$, for each of the subclassifiers, $f_{i}$ in
$\bar{f}$. We assume here that the hard-classifier ensemble and the soft
ensemble (that Smooth-Mean uses) are equivalent. Under this assumption, as
Smooth-Mean relies on majority vote, in order for ${\mathbf{x}}+\delta$ to be
an adversarial example, we need at least half of the classifiers to fail. To
analyse this, let $m_{i}$ be a Bernoulli random variable such that it takes
the value $1$ if classifier $f_{i}$ fails and $0$ otherwise.

Thus,

$\displaystyle{\mathbb{P}}[\|\delta\|<R_{i}]={\mathbb{P}}[m_{i}=1]=\alpha$

Therefore, for ${\mathbf{x}}+\delta$ to be an adversarial example,

$\displaystyle{\mathbb{P}}[\|\delta\|<R]$
$\displaystyle={\mathbb{P}}[\sum_{i=1}^{k}m_{i}\geq k/2]$

Using a Chernoff bound [30, Thm. 2.3.1] for the sum of independent Bernoulli
random variables, we get;

$\displaystyle{\mathbb{P}}[\sum_{i=1}^{k}m_{i}\geq k/2]\leq
e^{-k\alpha}(2e\alpha)^{k/2}$

Note that this function decays very quickly with $k$, and therefore can be
easily tuned to get better confidence bounds. ∎

While our approach relies on analysing a specific version of the adversarial
example which attacks all classifiers simultaneously, we recognize that this
might not be the case in general. For example, another attack may presume to
make the classifier abstain every time. We do not analyse this case here, and
leave the details to future work.

###  C.1 Analysing Logits under Smooth-Reduce Ensembling

We further validate our claims regarding confidence intervals of Smooth-Reduce
certificates by analysing the logit distribution for standard RS and Smooth-
Reduce classifiers.

Setup: We study the distributions of logits for the most probable and the
second most probable class for standard RS and Smooth-Reduce classifiers. For
this, we consider a few test datapoints for both images, and videos and
certify the best SmoothAdv classifier. Further, we certify both Smooth-Max and
Smooth-Mean classifiers under the same setup. We then plot histograms of the
distributions of logits. Fig. 11 and Fig. 12 show exemplars of generated
histograms.

Observations and Inferences. Notice that the certified radius, $R$ from Eq. 2
is proportional to the difference in the estimated probabilities of the two
most probable classes. This difference is also proportional to the classifier
margin. Therefore, in order to get better certificates, we need to ensure that
the smooth-classifier presents large margins, as well higher probability
estimate for the true class, $c_{A}$. Also, in order to reduce abstentions,
$\underline{p_{A}}>1/2$ and its variance must be low.

We see that Smooth-Max and Smooth-Mean outperform SmoothAdv on both these
criterion in Fig. 11. Notice here that $R$ depends on the difference between
the means of the distribution for the most probable class (blue) and the
second most probable class (orange). We see that while Smooth-Max outperforms
SmoothAdv in terms of the overall proabability estimate, the margin itself is
not improved much. This may lead to higher abstention rates as well as lower
certificates. However, Smooth-Mean showcases not only higher estimates of
$\underline{p_{A}}$ but also a lower variance, thus improving upon both the
certified radius and probability of abstention.

|  |
---|---|---
Figure 11: Logit Distributions for Smooth Classifiers for CIFAR-10.The
histograms are arranged as follows: (L) SmoothAdv classifier, (M) Smooth-Max
classifier, and (R) Smooth-Mean classifier. The blue bars the logits for the
most probable class, while the orange represent those for the second-most
probable class. For good smooth classifiers, the blue peak should be at $1.0$
with low variance and the orange peak should be close to $0$. Observe that
Smooth-Max is performs better than SmoothAdv on the first criterion, while
Smooth-Mean performs better on both.

For an exemplar certificate in the case of video classifiers, we immediately
observe similar behavior. In Fig. 12, we observe logit distributions for two
examples from the UCF-101 dataset. It is clearly evident that while Smooth-Max
and Smooth-Mean provide better margins and lower variance than standard
randomized smoothing. However, the logit values are still skewed lower than
those for images, and the variance across the logit values is fairly higher.
This explains why our video certificates are far lower than image
certificates. We conjecture that this is an effect of the difficulty in
training noise-robust 3D CNN models for video. However, we leave exploring
this phenomenon to future work.

|  |
---|---|---
|  |
Figure 12: Logit Distributions for Smooth Classifiers for UCF-101. (L) shows
logit distributions for standard randomized smoothing, (M) and (R) show the
same for Smooth-Max and Smooth-Mean respectively. Observe that in comparison
with image classifiers, RS and Smooth-Reduce with video classifiers leads to
lower $\underline{p_{A}}$ estimates as well as high variance. This results in
lower certified radii and higher abstention rates as seen in Fig. 6. However,
Smooth-Mean still outperforms SmoothAdv.

###  C.2 Some Additional Discussion on Confidence Intervals for Ensembling

We also reproduce some results by [11] which support increasing success rates
for Smooth-Mean.

Horvath  _et al._ [11] analyse the following soft ensemble classifier,

$\displaystyle\bar{f}({\mathbf{x}})=\frac{1}{k}\sum_{i=1}^{k}f_{i}({\mathbf{x}}).$

Let ${\mathbf{y}}_{i}$ be the logits from each of the sub-classifiers,
$f_{i}$, and ${\mathbf{y}}$ be the same for the ensembled classifier. They
further model ${\mathbf{y}}_{i}={\mathbf{y}}_{i,c}+{\mathbf{y}}_{i,p}$, where
${\mathbf{y}}_{i,c}$ is a random variable representing the contribution of the
$i^{th}$ sub-classifier and ${\mathbf{y}}_{i,p}$ represents the contribution
due to random noise added during randomized smoothing. Further they assume,
$\mathbb{E}[{\mathbf{y}}_{i,c}]={\mathbf{c}}$ and
$\mathbb{E}[{\mathbf{y}}_{i,p}]=0$. The variance of ${\mathbf{y}}_{p}$ is
assumed to be $\Sigma_{p}$ where $\Sigma_{ii}=\sigma_{i}^{2}$, and
$\Sigma_{ij}=\sigma_{i}\sigma_{j}\rho{ij}$. This holds as the two processes of
training and smoothing are independent. Notice that Smooth-Mean classifiers
are a special class of such classifiers, where the sub-classifiers are
constructed with independent sampling matrices.

Now, they analyse the class margins, $t_{i}=y_{1}-y_{1}$ where $y_{i}$ are
elements of ${\mathbf{y}}$ and $1$ is the majority class (WLOG). Notice,

$\displaystyle\mathbb{E}[{\mathbf{z}}_{i}]=c_{1}-c_{i}$
$\displaystyle\text{Var}[{\mathbf{t}}_{i}]=\sigma_{p,1}^{2}+\sigma_{p,i}^{2}+\sigma_{c,1}^{2}+\sigma_{c,i}^{2}-2\sigma_{p,1}\sigma_{p,i}\rho{p1,i}-2\rho{c,1i}\sigma+{c,1}\sigma{c,i}$

Through careful arithmetic, they show that,

$\text{Var}(\bar{{\mathbf{t}}})=\sigma_{p}^{2}(k)+\sigma_{c}^{2}(k),$

where $\sigma_{m}^{2}=\frac{k+{k\choose
2}\zeta_{m}}{k^{2}}(\sigma_{p,1}^{2}+\sigma_{p,i}^{2}-2\rho_{p,1i}\sigma{p,1}\sigma_{p,i}$
for $m\in[p,c]$, and $\zeta_{m}\in[0,1]$ refers to parameter denoting
covariance between $y_{i,m}$ and $y_{j,m}$; refer [11] for more details.

This decoupling of the variance between the perturbations due to RS and
ensembling proves to be important in understanding the benefits of ensembling.
They present the following result on success probabilities,

Informal Theorem[From [11]] For a soft-ensemble of $k$ classifiers which
provides a certificate with radius $R$ with probability $1-\alpha_{1}$, the
upper bound of the probability of failure decreases with $O(k^{2})$

To measure the effect on success probability, we consider the probability of a
majority of the sub-classifiers predicting class $1$, $\beta_{1}$.

$\beta_{1}={\mathbb{P}}(\bar{f}({\mathbf{x}}+{\mathbf{z}})=1)\\\
={\mathbb{P}}(\bar{{\mathbf{t}}}>0:\forall i\in[2,C])\\\
=\int_{\bar{{\mathbf{z}}}>0:\forall
i\in[2,C]}{\mathbb{P}}(\bar{{\mathbf{t}}}).d\bar{{\mathbf{z}}}$

While this integral cannot be evaluated directly as we do not know the density
function for ${\mathbf{z}}$, we can lower bound $\beta_{1}$ using Chebyshev’s
inequality and the union bound over the incorrect $[2,C]$ classes.

$\beta_{1}\geq
1-\sum_{i=1}^{C}\frac{(\sigma_{i,c}(k)^{2}+\sigma_{i,p}(k)^{2}}{(c_{1}-c_{i})^{2}}$

. As $\sigma_{i,c}$ and $\sigma_{i,k}$ decrease quadratically with increasing
$k$, we can prove the above theorem.


# Gabor is Enough: Interpretable Deep Denoising with a Gabor Synthesis
Dictionary Prior

Nikola Janjušević, Amirhossein Khalilian-Gourtani, and Yao Wang  New York
University, Tandon School of Engineering, Brooklyn, USA
Email<EMAIL_ADDRESS>
###### Abstract

Image processing neural networks, natural and artificial, have a long history
with orientation-selectivity, often described mathematically as Gabor filters.
Gabor-like filters have been observed in the early layers of CNN classifiers
and even throughout low-level image processing networks. In this work, we take
this observation to the extreme and explicitly constrain the filters of a
natural-image denoising CNN to be learned 2D real Gabor filters. Surprisingly,
we find that the proposed network (GDLNet) can achieve near state-of-the-art
denoising performance amongst popular fully convolutional neural networks,
with only a fraction of the learned parameters. We further verify that this
parameterization maintains the noise-level generalization (training vs.
inference mismatch) characteristics of the base network, and investigate the
contribution of individual Gabor filter parameters to the performance of the
denoiser. We present positive findings for the interpretation of dictionary
learning networks as performing accelerated sparse-coding via the importance
of untied learned scale parameters between network layers. Our network’s
success suggests that representations used by low-level image processing CNNs
can be as simple and interpretable as Gabor filterbanks.

##  I Introduction and Background

In recent years, deep neural network (DNN) and convolutional neural network
(CNN) architectures have claimed state-of-the-art performance on low-level
image processing tasks such as denoising[1, 2, 3]. However, most DNNs are
constructed as black-box function approximators whose architectures are
largely derived from trial and error [4], ignoring the theoretical signal
processing backgrounds related to the task at hand. Such networks have been
shown to suffer a catastrophic performance failure when a mismatch is present
between the training and inference scenarios [5, 6]. More recently, there has
been a growing desire for principled approach to DNN design that leads to
better analysis and possible improvement of the network[6, 7, 8, 9, 10, 11,
12]. To this end, in [6] we introduced an architecture (CDLNet) derived from
direct parameterization of an iterative algorithm solving the convolutional
dictionary learning (CDL) problem. By relating the thresholds in CDLNet to the
input image noise-level, this network showed near-perfect generalization at
inference to noise-levels unseen during training. Notably, CDLNet achieves
similar performance to other state-of-the-art fully convolutional neural
networks (FCNNs)[1, 2] while side-stepping the use of feature-domain
processing and limiting the representation to only analysis and synthesis
convolutional dictionaries. Such directly-parameterized dictionary learning
based networks [6, 8, 13, 10] challenge the notion of low-level image
processing CNNs capturing complex signal representations in hidden feature
domains, and suggest that much simpler mechanisms, such as subband
representations, can account for their performance.

Many dictionary learning frameworks for low-level image processing tasks,
especially denoising, learn “Gabor-like” dictionaries[6, 8, 14].
Neuroscientific investigations suggest that V1 simple cells in primary visual
cortex behave like Gabor filters [15, 16, 17]. The early visual cortex
responds maximally to lines or edges with a specific orientation which can be
achieved with Gabor filter with a proper scale, a specific direction and a
spatial frequency [18]. More recently, Gabor filters have attracted attention
in CNN design, from preprocessing stage [19], to use of Gabor filters in early
CNN layer [20], to combining Gabor and learned convolutional filters [21].
Fixed Gabor filters have also been shown to reduce the computational
complexity of CNNs, however, with a significant performance reduction [22].
These works overall suggest that use of Gabor filters in CNN design leads to
better performance, albeit with a more complex network, or reduced complexity
with degraded performance.

The goal of this work is to quantitatively explore the Gabor-like behavior of
learned filters in convolutional dictionary based neural networks and pose the
question “can Gabor-like be replaced with Gabor?”. We tackle this question by
replacing the filters of CDLNet [6] with parameterized 2D Gabor functions. In
place of learned filters, this network learns sets of parameters that generate
Gabor filters (Section II-B). Surprisingly, we find that this representation
is sufficient for achieving near state-of-the-art performance in the natural-
image denoising task (Section III-B) with respect to popular FCNNs such as
DnCNN [1] and FFDNet [2]. We verify that the near-perfect denoising
generalization of CDLNet to unseen noise-levels are retained with this
parameterization (Section III-C). Finally, in [6] we demonstrated the
importance of untying filter weights between network layers. With the proposed
Gabor filterbank representation, we further explore untying of individual
parameters of the Gabor function between layers and their contribution to
denoising performance(Section III-E).

Figure 1: Left: Block diagram of GDLNet, CDLNet with mixture of Gabor (MoG)
filterbanks. Analysis and synthesis convolutions map from $1\rightarrow M$ and
$M\rightarrow 1$ channels, respectively. We emphasize that GDLNet/CDLNet do
not process signals in a “learned feature domain” in contrast to the multi-
channel filtering ($M\rightarrow M$ channels) in popular DNNs such as DnCNN
[1]. Right: Mixture of Gabor filters are generated as a sum of parameterized
Gabor filters.

##  II Problem Statement and Proposed Method

In this manuscript, we look at the denoising of natural images via
convolutional neural networks (CNNs). We denote our observations contaminated
with additive Gaussian white noise (AWGN) , $\bm{y}=\bm{x}+\bm{\nu}$, where
$\bm{\nu}\sim\mathcal{N}(0,\sigma\boldsymbol{I})$.

###  II-A CDLNet Architecture

The Convolutional Dictionary Learning Network (CDLNet) architecture [6] is a
denoising neural network derived from the Iterative Shrinkage Thresholding
Algorithm (ISTA)[23]. Much like a traditional CNN, it employs layered
convolutions and point-wise nonlinearities, as described below,

$\displaystyle\hat{\bm{x}}$
$\displaystyle=\bm{D}\bm{z}^{(K)},\quad\bm{z}^{(0)}=\bm{0},\quad
k=0,1,\dots,K-1,$ (1) $\displaystyle\bm{z}^{(k+1)}$
$\displaystyle=\operatorname{\mathrm{ST}}\left(\bm{z}^{(k)}-{\bm{A}^{(k)}}^{T}(\bm{B}^{(k)}\bm{z}^{(k)}-\bm{y}),\,\boldsymbol{\tau}^{(k)}\right).$

Here, ${\bm{A}^{(k)}}^{T},\bm{B}^{(k)}$ refer to $M$-subband analysis and
synthesis filterbank convolutions, respectively, with stride-$s$,
$\mathrm{ST}(\bm{x},\tau)[n]\coloneqq\operatorname{sign}(\bm{x}[n])\max(0,\lvert\bm{x}[n]\rvert-\tau)$
is known as the element-wise soft-thresholding operator, and
$\bm{\tau}^{(k)}\in\mathbb{R}^{M}$ are the networks learned non-negative
thresholds. The final denoised image $\hat{\bm{x}}$ is given by convolution
with the network’s synthesis filterbank (dictionary), $\bm{D}$.

CDLNet’s interpretation as an accelerated sparse-coding algorithm allows for
consideration of a varying input noise-level via augmenting the thresholds of
the network. In [6], it was shown that the following noise-adaptive
thresholding,

$\boldsymbol{\tau}^{(k)}=\boldsymbol{\tau}^{(k)}_{0}+\boldsymbol{\tau}^{(k)}_{1}\sigma,\quad\mathrm{such~{}that}\quad\boldsymbol{\tau}^{(k)}_{\\{0,1\\}}\geq
0,$ (2)

with $\boldsymbol{\tau}^{(k)}_{\\{0,1\\}}\in\mathbb{R}^{M}$ learned
parameters, yields a near perfect generalization to input noise-levels outside
the network’s training range at inference time. This behavior is in contrast
to the catastrophic failures of black box DNNs observed outside their training
ranges [6, 5].

###  II-B Parameterized Gabor Filterbanks

The learned dictionary elements in[6] are “Gabor-like” filters. In this work,
we consider the explicit parameterization of CDLNet’s filterbanks (both
analysis and synthesis) as learnable 2D real Gabor filters,

$g(\bm{x};\phi)=\alpha
e^{-\lVert\bm{a}\circ\bm{x}\rVert_{2}^{2}}\cos(\boldsymbol{\omega}_{0}^{T}\bm{x}+\psi),$
(3)

with $\bm{x}\in\mathbb{R}^{2}$ denoting spatial position, $\circ$ denoting
element-wise product, and
$\phi=(\alpha,\bm{a},\bm{\omega}_{0},\psi)\in\mathbb{R}^{6}$. We refer to
these as $\alpha\in\mathbb{R}$ the scale, $\bm{a}\in\mathbb{R}^{2}$ the (root)
precision, $\bm{\omega}_{0}\in\mathbb{R}^{2}$ the frequency, and
$\psi\in\mathbb{R}$ the phase parameters. These filters are orientation
selective, as seen in their Fourier domain representation of conjugate
symmetric Gaussians centered at frequency $\pm\bm{\omega}_{0}$,

$\mathcal{F}\\{g(;\phi)\\}(\bm{\omega})=\frac{\alpha}{2}\left(e^{j\psi}e^{-\lVert\frac{\bm{\omega}-\bm{\omega}_{0}}{2\pi\bm{a}}\rVert_{2}^{2}}+e^{-j\psi}e^{-\lVert\frac{\bm{\omega}+\bm{\omega}_{0}}{2\pi\bm{a}}\rVert_{2}^{2}}\right).$

To further enable these filters to admit a sparse image representation, we
consider parameterizing the filters of CDLNet as a mixture of Gabor (MoG)
filters,

$d(\bm{x};\bm{\phi})=\sum_{i=1}^{S}g(\bm{x};\bm{\phi}_{i}).$ (4)

A single MoG filter of order $S>0$ is parameterized by
$\bm{\phi}\in\mathbb{R}^{6S}$, and a MoG $M$-subband analysis/synthesis
filterbank is parameterized by
$\Phi=[\bm{\phi}^{1},\dots,\bm{\phi}^{M}]\in\mathbb{R}^{6SM}$. This is in
contrast to the unconstrained filters in CDLNet, parameterized in
$\mathbb{R}^{PM}$ for $\sqrt{P}\times\sqrt{P}$ filters.

The MoG representation allows for a wider class of filters to be learned with
the same computational complexity in the forward pass of the network, while
maintaining the interpretablility of the Gabor parameterization over
unconstrained filters. In the digital realization of these Gabor filters, we
limit their spatial extent to $\sqrt{P}\times\sqrt{P}$ and evaluate (3) at
lattice points. Interestingly, the spatial extent of these filters has no
effect on their parameter count. We refer to the CDLNet architecture with
parameterized mixture of Gabor filters as the Gabor Dictionary Learning
Network (GDLNet). Its parameters are given by
$\Theta=(\\{\Phi_{A}^{(k)},\Phi_{B}^{(k)},\bm{\tau}^{(k)}\\}_{k=0}^{K-1},\Phi_{D})$,
where $\Phi_{A}^{(k)},\Phi_{B}^{(k)},$ refer to the Gabor parameters of the
analysis, synthesis filterbanks of the $k$th layer, and similarly $\Phi_{D}$
refers to the Gabor parameters of the synthesis dictionary $\bm{D}$ (see (1)).
Figure 1 shows an overview of the GDLNet architecture with MoG filterbanks.

##  III Experimental Results

###  III-A Training and Inference Setup

We follow the training and inference setup of CDLNet [6]. All models are
trained on the BSD432 dataset [24] under supervised learning with the mean-
squared-error loss,

$\underset{\begin{subarray}{c}\Phi_{A}^{(k)},~{}\Phi_{B}^{(k)},~{}\Phi_{D},\\\
\boldsymbol{\tau}^{(k)}\geq
0\end{subarray}}{\mathrm{minimize}}\quad\sum_{(\bm{y}_{i},\bm{x}_{i})\in\mathcal{D}}\|\bm{x}_{i}-\bm{D}\bm{z}^{(K)}(\bm{y}_{i})\|_{2}^{2},$
(5)

where $(\bm{x}_{i},\bm{y}_{i})\in\mathcal{D}$ represent ground-truth and
observed images of the dataset. Data-augmentation, as well as additive noise,
are applied online. Noise-level $\sigma\in\sigma^{\mathrm{train}}$ is sampled
uniformly for each mini-batch sample. Noise-level adaptive models are given
the ground truth noise-level at both training and inference, though common
noise-level estimators [25, 26] may be applied instead at inference (see [6]
for details).

Architectures: Suffixes (-S,-B) are used to denote models trained on a single
noise-level (-S) or over a noise-level range without adaptive thresholds (-B).
Unless otherwise specified, table I shows the hyperparameters of our proposed
GDLNet. Code available at [27].

TABLE I: Architectures of GDLNet models presented in the experiments section.
$K$ layers, $M$ subbands, stride-$s$, filter-size $P$, MoG order $S$.
Parameters are as shown unless otherwise specified.

Name | $K$ | $M$ | $s$ | $P$ | $S$ | Params
---|---|---|---|---|---|---
CDLNet(-S) [6] | 30 | 169 | 2 | $7\times 7$ | n/a | 507k
GDLNet(-S) (MoG 1) | 30 | 169 | 2 | $11\times 11$ | 1 | 66k
GDLNet(-S,-B) (MoG 3) | 30 | 169 | 2 | $11\times 11$ | 3 | 188k

Initialization: As in [6], all filterbanks are initialized with the same
parameters and subsequently normalized by the their spectral norm,
$L=\lVert\bm{D}^{T}\bm{D}\rVert_{2}$. For the Gabor filters, this
normalization takes place via the scale parameter,
$\alpha\leftarrow\alpha/\sqrt{L}$.

Gabor Filters: The Gabor filter parameterization requires that we recompute
the filters from their parameters after each gradient update of the optimizer.
This offers a minor computational overhead during training, however the filter
weights may be computed and stored for faster inference once training is
complete.

###  III-B Single Noise-level Denoising

Single noise-level grayscale denoising results, as compared to state-of-the-
art popular FCNNs, are shown in Table II. The non-learned BM3D method [28] is
given as baseline. GDLNet-S (MoG 1) performs competitively in the low
parameter count regime, outperforming CSCNet [8] and small-CDLNet-S [6]. The
use of higher order MoG filters in GDLNet-S (MoG 3) allows it to perform on
par with DnCNN [1] and FFDNet [2] while using less than half the learned
parameters.

TABLE II: Grayscale image denoising performance (PSNR) on BSD68 testset
($\sigma=\sigma^{\mathrm{train}}=\sigma^{\mathrm{test}}$). All learned models
trained on BSD432[24]. † trained on CBSD432 $+$ Waterloo ED [29].

Model | Params | Noise-level ($\sigma$)
---|---|---
15 | 25 | 50
BM3D [28] | - | 31.07 | 28.57 | 25.62
CSCNet [8]† | 64k | 31.57 | 29.11 | 26.24
small-CDLNet-S [6] | 64k | 31.60 | 29.11 | 26.19
GDLNet-S (MoG 1) | 66k | 31.59 | 29.13 | 26.21
FFDNet [2] | 485k | 31.63 | 29.19 | 26.29
DnCNN [1] | 556k | 31.72 | 29.22 | 26.23
CDLNet-S [6] | 507k | 31.74 | 29.26 | 26.35
GDLNet-S (MoG 3) | 188k | 31.68 | 29.22 | 26.30

MoG order and filter-size hyperparameters used in Table II are verified in
Tables III and IV. An important boost in performance is given by MoG orders
$>1$, though diminishing returns are observed above order 3. Minimal
performance gains are observed as the filter size increases beyond $11\times
11$.

TABLE III: Effect of mixture of Gabor (MoG) order for GDLNet-S on denoising
performance (PSNR) over BSD68 testset [24]
($\sigma=\sigma^{\mathrm{train}}=\sigma^{\mathrm{test}}$).

MoG order | Params | Noise-level ($\sigma$)
---|---|---
15 | 25 | 50
1 | 66k | 31.59 | 29.13 | 26.21
2 | 127k | 31.65 | 29.18 | 26.27
3 | 188k | 31.68 | 29.22 | 26.30
4 | 248k | 31.70 | 29.23 | 26.31
5 | 309k | 31.70 | 29.24 | 26.33

TABLE IV: Effect of filter size for GDLNet-S (MoG 1) on denoising performance
(PSNR) over grayscale Kodak dataset [30]
($\sigma=\sigma^{\mathrm{train}}=\sigma^{\mathrm{test}}$).

Filter size (P) | Noise-level ($\sigma$)
---|---
15 | 25 | 50
$3\times 3$ | 32.45 | 30.06 | 27.21
$7\times 7$ | 32.55 | 30.17 | 27.36
$11\times 11$ | 32.57 | 30.21 | 27.40
$15\times 15$ | 32.58 | 30.22 | 27.38

###  III-C Blind Denoising and Generalization

We consider the blind denoising and generalization scenarios, and compare
GDLNet models of different MoG order equipped with adaptive thresholds. In
Figures 2a and 2b, we show the performance of the models trained on the noise
range $\sigma^{\mathrm{train}}=[1,20]$ and $[20,30]$, respectively, and tested
on different noise-levels $\sigma^{\mathrm{test}}\in[5,50]$ for grayscale
images. The blind denoising version of DnCNN [1], DnCNN-B, and FFDNet trained
on $\sigma^{\mathrm{train}}=[01,20]$ and $[20,30]$, are as reported in [6].
The single points on these plots show the performance of GDLNet-S (MoG 3)
model with single noise-level training (i.e.
$\sigma^{\mathrm{train}}=\sigma^{\mathrm{test}}$). As shown in Figures 2a and
2b, all networks perform closely over the training noise-range. Conversely, we
observe the catastrophic failure of models without adaptation when
generalizing outside the training noise-level. GDLNet nearly matches the
performance of the models trained for a specific noise-level (GDLNet-S) across
the test range. We observe a slight improved performance for GDLNet with
higher order. Note that the CDLNet model architecture disentangles the noise
generalization from the dictionary structure. Consequently, GDLNets of
different order show similar generalization capability when equipped with
adaptive thresholds. Visual comparisons in Figures 2c and 2d show superior
performance of GDLNet compared to other models.

(a) PSNR plot $\sigma^{\mathrm{train}}={[01,20]}$

(b) PSNR plot $\sigma^{\mathrm{train}}={[20,30]}$

(c) Visual comparison for $\sigma^{\mathrm{train}}={[01,20]}$ and
$\sigma^{\mathrm{test}}={50}$

(d) Visual comparison for $\sigma^{\mathrm{train}}={[20,30]}$ and
$\sigma^{\mathrm{test}}={50}$

Figure 2: (a,b) Performance of different denoising networks trained on
$\sigma^{\mathrm{train}}$ and tested on $\sigma^{\mathrm{test}}$. Average PSNR
calculated over BSD68 [24]. (c,d) Visual comparison of different networks
tested on noise-level $\sigma^{\mathrm{test}}=50$. GDLNet is MoG order 3.
Details are better visible by zooming.

###  III-D Comparison of Learned Dictionaries

Figure 3 shows the final synthesis dictionaries of CDLNet, GDLNet, and
GDLNet’s initial dictionary. The use of parameterized Gabor filters adds to
the interpretability of GDLNet as we can understand the subband
representations simply as (a sum of) orientation features. GDLNet has cleaner
looking filters and does not exhibit phase-shifted copies of filters, as in
seen CDLNet Fig. 3a. This is because each filter in GDLNet is centered, by
construction. As shown in Fig. 3a, dictionary in GDLNet MoG order 3 exhibits
complex structures that are not simply a single orientation, showing that the
additional flexibility of the MoG representation is contributing to the
performance gain. The random initialization of Gabor parameters $\phi$ yields
a good starting point for the dictionary (Fig. 3d) and the network is free to
learn different orientations and precisions as desired (Fig. 3c).

Figure 3: Synthesis dictionary ($\bm{D}$) filters for (a) CDLNet [6], (b)
GDLNet with MoG order 3, (c) GDLNet with MoG order 1, and (d) initial Gabor
dictionary of (c). All models trained on $\sigma^{\mathrm{train}}=[20,30]$.
Filters are ordered by relative usage over the BSD68 dataset [24].

###  III-E Ablation Study of Gabor Filterbanks

The success of CDLNet, in contrast to CSCNet [8] and others [31, 9], is
largely attributed to untying of the weights between layers of the network.
However, CDLNet and other directly parameterized unrolled networks [8, 7, 9,
31] are often interpreted as performing an accelerated sparse coding, either
via a learned preconditioning and/or learned step-sizes of the original
algorithm [32, 33]. Untying the weights between layers makes this
interpretation more difficult as the network is able to vary arbitrarily from
layer-to-layer.

GDLNet allows us to examine the relationship of weight tying more closely by
only coupling certain parameters of the Gabor filters. Figure 4 shows the
performance of GDLNet as we progressively untie parameters between layers.
While untying the phase parameter ($\psi$) shows increased performance
compared to baseline, very significant increase is seen by simply untying the
scale parameter ($\alpha$). To the benefit of the sparse-coding interpretation
of the network, the scale parameter ($\alpha$) corresponds most closely to
having learned step-sizes within the original algorithm. The minimal
performance improvement observed when phase and scale are untied further
verifies the sparse-coding interpretation of the network. In contrast, untying
the phase and frequency alone does not yield as great of an improvement. This
can be justified by considering that phase and frequency parameters may have a
considerable effect on the overall magnitude of the filter, and thus can
indirectly effect the implicit learned step-sizes.

Figure 4: Effect of untied parameters between layers $k=0,1,\dots,K-1$ on
performance of GDLNet (MoG 1). PSNR averaged over BSD68 [24],
$\sigma^{\mathrm{train}}=\sigma^{\mathrm{test}}=25$. Refer to (3) for
description of Gabor filter parameters.

##  IV Conclusion

In this work, we replaced the filters of an unrolled dictionary learning
network (CDLNet) with parameterized 2D real Gabor functions. By using a
mixture of Gabor filters, we achieved near state-of-the-art denoising
performance at a fraction of parameter count and retained the network’s
generalization behavior in unseen noise-levels. Progressively untying the
Gabor filter parameters allowed us to understand the contribution of each
component to denoising performance and strengthened the sparse-coding
interpretation of our network. In future, we aim to further investigate the
use of Gabor representation in imaging inverse problems. The simplicity of
this representation is suited for a better understanding of the network and
may also be exploited for novel capabilities.

## References

  * [1] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising,” _IEEE Transactions on Image Processing_ , vol. 26, no. 7, p. 3142–3155, 2017.
  * [2] K. Zhang, W. Zuo, and L. Zhang, “FFDNet: Toward a fast and flexible solution for CNN-based image denoising,” _IEEE Transactions on Image Processing_ , vol. 27, no. 9, p. 4608–4622, 2018.
  * [3] H. Zheng, H. Yong, and L. Zhang, “Deep convolutional dictionary learning for image denoising,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021, pp. 630–641.
  * [4] V. Buhrmester, D. Münch, and M. Arens, “Analysis of explainers of black box deep neural networks for computer vision: A survey,” _arXiv preprint arXiv:1911.12116_ , 2019.
  * [5] S. Mohan, Z. Kadkhodaie, E. P. Simoncelli, and C. Fernandez-Granda, “Robust and interpretable blind image denoising via bias-free convolutional neural networks,” in _International Conference on Learning Representations (ICLR)_ , 2020.
  * [6] N. Janjušević, A. Khalilian-Gourtani, and Y. Wang, “CDLNet: Noise-adaptive convolutional dictionary learning network for blind denoising and demosaicing,” _arXiv preprint arXiv:2112.00913_ , 2021.
  * [7] B. Lecouat, J. Ponce, and J. Mairal, “Fully trainable and interpretable non-local sparse models for image restoration,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [8] D. Simon and M. Elad, “Rethinking the CSC model for natural images,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 2274–2284.
  * [9] M. Scetbon, M. Elad, and P. Milanfar, “Deep k-svd denoising,” _ArXiv_ , vol. abs/1909.13164, 2019.
  * [10] S. Ravishankar, I. Y. Chun, and J. A. Fessler, “Physics-driven deep training of dictionary-based algorithms for mr image reconstruction,” in _2017 51st Asilomar Conference on Signals, Systems, and Computers_ , 2017, pp. 1859–1863.
  * [11] B. Malézieux, T. Moreau, and M. Kowalski, “Dictionary and prior learning with unrolled algorithms for unsupervised inverse problems,” _arXiv preprint arXiv:2106.06338_ , 2021.
  * [12] E. Kobler, A. Effland, K. Kunisch, and T. Pock, “Total deep variation for linear inverse problems,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [13] B. Lecouat, J. Ponce, and J. Mairal, “A flexible framework for designing trainable priors with adaptive smoothing and game encoding,” in _Advances in Neural Information Processing Systems_ , vol. 33, 2020, pp. 15 664–15 675.
  * [14] E. P. Simoncelli and B. A. Olshausen, “Natural image statistics and neural representation.” _Annual review of neuroscience_ , vol. 24, pp. 1193–216, 2001.
  * [15] J. P. Jones and L. A. Palmer, “An evaluation of the two-dimensional gabor filter model of simple receptive fields in cat striate cortex,” _Journal of neurophysiology_ , vol. 58, no. 6, pp. 1233–1258, 1987.
  * [16] S. Marĉelja, “Mathematical description of the responses of simple cortical cells,” _JOSA_ , vol. 70, no. 11, pp. 1297–1300, 1980.
  * [17] J. G. Daugman, “Two-dimensional spectral analysis of cortical receptive field profiles,” _Vision research_ , vol. 20, no. 10, pp. 847–856, 1980.
  * [18] N. Kruger, P. Janssen, S. Kalkan, M. Lappe, A. Leonardis, J. Piater, A. J. Rodriguez-Sanchez, and L. Wiskott, “Deep hierarchies in the primate visual cortex: What can we learn for computer vision?” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 35, no. 8, pp. 1847–1871, 2012\.
  * [19] H. Yao, L. Chuyi, H. Dan, and Y. Weiyu, “Gabor feature based convolutional neural network for object recognition in natural scene,” in _2016 3rd International Conference on Information Science and Control Engineering (ICISCE)_. IEEE, 2016, pp. 386–390.
  * [20] J. Bai, Y. Zeng, Y. Zhao, and F. Zhao, “Training a v1 like layer using gabor filters in convolutional neural networks,” in _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2019, pp. 1–8.
  * [21] S. Luan, C. Chen, B. Zhang, J. Han, and J. Liu, “Gabor convolutional networks,” _IEEE Transactions on Image Processing_ , vol. 27, no. 9, pp. 4357–4366, 2018.
  * [22] S. S. Sarwar, P. Panda, and K. Roy, “Gabor filter assisted energy efficient fast learning convolutional neural networks,” in _2017 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED)_. IEEE, 2017, pp. 1–6.
  * [23] A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” _SIAM Journal on Imaging Sciences_ , vol. 2, pp. 183–202, 01 2009.
  * [24] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in _Proceedings of 8th IEEE International Conference on Computer Vision (ICCV)_ , vol. 2, 2001, pp. 416–423.
  * [25] S. G. Chang, B. Yu, and M. Vetterli, “Adaptive wavelet thresholding for image denoising and compression,” _IEEE Transactions on Image Processing_ , vol. 9, no. 9, pp. 1532–1546, 2000.
  * [26] X. Liu, M. Tanaka, and M. Okutomi, “Single-image noise level estimation for blind denoising,” _IEEE Transactions on Image Processing_ , vol. 22, no. 12, pp. 5226–5237, Dec. 2013.
  * [27] N. Janjušević, _CDLNet repository_ , 2021. [Online]. Available: https://github.com/nikopj/CDLNet-OJSP
  * [28] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” _IEEE Transactions on Image Processing_ , vol. 16, no. 8, pp. 2080–2095, 2007.
  * [29] K. Ma, Z. Duanmu, Q. Wu, Z. Wang, H. Yong, H. Li, and L. Zhang, “Waterloo Exploration Database: New challenges for image quality assessment models,” _IEEE Transactions on Image Processing_ , vol. 26, no. 2, pp. 1004–1016, Feb. 2017.
  * [30] R. Franzen, _The Kodak Color Image Dataset,_. [Online]. Available: http://r0k.us/graphics/kodak/
  * [31] H. Sreter and R. Giryes, “Learned convolutional sparse coding,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 2191–2195.
  * [32] P. Ablin, T. Moreau, M. Massias, and A. Gramfort, “Learning step sizes for unfolded sparse coding,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 13 100–13 110.
  * [33] X. Chen, J. Liu, Z. Wang, and W. Yin, “Theoretical linear convergence of unfolded ISTA and its practical weights and thresholds,” in _Advances in Neural Information Processing Systems_ , vol. 31, 2018.