Normalization-Equivariant Neural Networks with Application to Image
  Denoising
In many information processing systems, it may be desirable to ensure that
any change of the input, whether by shifting or scaling, results in a
corresponding change in the system response. While deep neural networks are
gradually replacing all traditional automatic processing methods, they
surprisingly do not guarantee such normalization-equivariance (scale + shift)
property, which can be detrimental in many applications. To address this issue,
we propose a methodology for adapting existing neural networks so that
normalization-equivariance holds by design. Our main claim is that not only
ordinary convolutional layers, but also all activation functions, including the
ReLU (rectified linear unit), which are applied element-wise to the
pre-activated neurons, should be completely removed from neural networks and
replaced by better conditioned alternatives. To this end, we introduce
affine-constrained convolutions and channel-wise sort pooling layers as
surrogates and show that these two architectural modifications do preserve
normalization-equivariance without loss of performance. Experimental results in
image denoising show that normalization-equivariant neural networks, in
addition to their better conditioning, also provide much better generalization
across noise levels.
SÃ©bastien Herbreteau, Emmanuel Moebel, Charles Kervrann
# Normalization-Equivariant Neural Networks with Application to Image Denoising

###### Abstract

In many information processing systems, it may be desirable to ensure that any change of the input, whether by shifting or scaling, results in a corresponding change in the system response. While deep neural networks are gradually replacing all traditional automatic processing methods, they surprisingly do not guarantee such normalization-equivariance (scale + shift) property, which can be detrimental in many applications. To address this issue, we propose a methodology for adapting existing neural networks so that normalization-equivariance holds by design. Our main claim is that not only ordinary convolutional layers, but also all activation functions, including the ReLU (rectified linear unit), which are applied element-wise to the pre-activated neurons, should be completely removed from neural networks and replaced by better conditioned alternatives. To this end, we introduce affine-constrained convolutions and channel-wise sort pooling layers as surrogates and show that these two architectural modifications do preserve normalization-equivariance without loss of performance. Experimental results in image denoising show that normalization-equivariant neural networks, in addition to their better conditioning, also provide much better generalization across noise levels.

## 1 Introduction

Sometimes wrongly confused with the invariance property which designates the characteristic of a function \(f\) not to be affected by a specific transformation \(\mathcal{T}\) applied beforehand, the equivariance property, on the other hand, means that \(f\) reacts in accordance with \(\mathcal{T}\). Formally, invariance is \(f\circ\mathcal{T}=f\) whereas equivariance reads \(f\circ\mathcal{T}=\mathcal{T}\circ f\), where \(\circ\) denotes the function composition operator. Both invariance and equivariance play a crucial role in many areas of study, including physics, computer vision, signal processing and have recently been studied in various settings for deep-learning-based models [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14].

In this paper, we focus on the equivariance of neural networks \(f_{\theta}\) to a specific transformation \(\mathcal{T}\), namely normalization. Although highly desirable in many applications and in spite of its omnipresence in machine learning, current neural network architectures do not equivary to normalization. With application to image denoising, for which _normalization-equivariance_ is generally guaranteed for a lot of conventional methods [18, 20, 27, 22], we propose a methodology for adapting existing neural networks, and in particular denoising CNNs [30, 32, 31, 34, 33], so that _normalization-equivariance_ holds by design. In short, the proposed adaptation is based on two innovations:

1. affine convolutions: the weights from one layer to each neuron from the next layer, _i.e._ the convolution kernels in a CNN, are constrained to encode affine combinations of neurons (the sum of the weights is equal to 1).
2. channel-wise sort pooling: all activation functions that apply element-wise, such as the ReLU, are substituted with higher-dimensional nonlinearities, namely two by two sorting along channels that constitutes a fast and efficient _normalization-equivariant_ alternative.
Despite strong architectural constraints, we show that these simple modifications do not degrade performance and, even better, increase robustness to noise levels in image denoising both in practice and in theory.

## 2 Related Work

A non-exhaustive list of application fields where equivariant neural networks were studied includes graph theory, point cloud analysis and image processing. Indeed, graph neural networks are usually expected to equivary, in the sense that a permutation of the nodes of the input graph should permute the output nodes accordingly. Several specific architectures were investigated to guarantee such a property [5; 6; 7]. In parallel, rotation and translation-equivariant networks for dealing with point cloud data were proposed in a recent line of research [12; 13; 14]. A typical application is the ability for these networks to produce direction vectors consistent with the arbitrary orientation of the input point clouds, thus eliminating the need for data augmentation. Finally, in the domain of image processing, it may be desirable that neural networks produce outputs that equivary with regard to rotations of the input image, whether these outputs are vector fields [8], segmentation maps [9; 10], labels for image classification [9] or even bounding boxes for object tracking [11].

In addition to their better conditioning, equivariant neural networks by design are expected to be more robust to outliers. A spectacular example has been revealed by S. Mohan _et al._[1] in the field of image denoising. By simply removing the additive constant ("bias") terms in neural networks with ReLU activation functions, they showed that a much better generalization at noise levels outside the training range was ensured. Although they do not fully elucidate why biases prevent generalization, and their removal allows it, the authors establish some clues that the answer is probably linked to the _scale-equivariant_ property of the resulting encoded function: rescaling the input image by a positive constant value rescales the output by the same amount.

## 3 Overview of normalization equivariance

### Definitions and properties of three types of fundamental equivariances

We start with formal definitions of the different types of equivariances studied in this paper.

**Definition 1**: _A function \(f:\mathbb{R}^{n}\mapsto\mathbb{R}^{m}\) is said to be:_

* _scale-equivariant if_ \(\forall x\in\mathbb{R}^{n},\forall\lambda\in\mathbb{R}^{+}_{*},\;f(\lambda x)= \lambda f(x)\,,\)__
* _shift-equivariant if_ \(\forall x\in\mathbb{R}^{n},\forall\mu\in\mathbb{R},\;f(x+\mu)=f(x)+\mu\,,\)__
* _normalization-equivariant if it is both scale-equivariant and shift-equivariant:_ \[\forall x\in\mathbb{R}^{n},\forall\lambda\in\mathbb{R}^{+}_{*},\forall\mu \in\mathbb{R},\;f(\lambda x+\mu)=\lambda f(x)+\mu\,,\]

_where addition with the scalar shift \(\mu\) is applied element-wise._

Note that the _scale-equivariance_ property is more often referred to as positive homogeneity in pure mathematics. Like linear maps that are completely determined by their values on a basis, the above described equivariant functions are actually entirely characterized by the values their take on specific subset of \(\mathbb{R}^{n}\), as stated by the following proposition (see proof in Appendix D.1).

**Proposition 1** (Characterizations): \(f:\mathbb{R}^{n}\mapsto\mathbb{R}^{m}\) _is entirely determined by its values on the:_

* _unit sphere_ \(\mathcal{S}\) _of_ \(\mathbb{R}^{n}\) _if it is scale-equivariant,_
* _orthogonal complement of_ \(\operatorname{Span}(\mathbf{1}_{n})\)_, i.e._ \(\operatorname{Span}(\mathbf{1}_{n})^{\perp}\)_, if it is shift-equivariant,_
* _intersection_ \(\mathcal{S}\cap\operatorname{Span}(\mathbf{1}_{n})^{\perp}\) _if it is normalization-equivariant,_

_where \(\mathbf{1}_{n}\) denotes the all-ones vector of \(\mathbb{R}^{n}\)._

Finally, Proposition 2 highlights three basic equivariance-preserving mathematical operations that can be used as building blocks for designing neural network architectures.

**Proposition 2** (Operations preserving equivariance): _Let \(f\) and \(g\) be two equivariant functions of the same type (either in scale, shift or normalization). Then, subject to dimensional compatibility, all of the following functions are still equivariant:_

* \(f\circ g\) _(_\(f\) _composed with_ \(g\)_),_
* \(x\mapsto(f(x)^{\top}\,g(x)^{\top}\,)^{\top}\) _(concatenation of_ \(f\) _and_ \(g\)_),_
* \((1-t)f+tg\) _for all_ \(t\in\mathbb{R}\) _(affine combination of_ \(f\) _and_ \(g\)_)._
### Examples of normalization-equivariant conventional denoisers

A ("blind") denoiser is basically a function \(f:\mathbb{R}^{n}\mapsto\mathbb{R}^{n}\) which, given a noisy image \(y\in\mathbb{R}^{n}\), tries to map the corresponding noise-free image \(x\in\mathbb{R}^{n}\). Since scaling up an image by a positive factor \(\lambda\) or adding it up a constant shift \(\mu\) does not change its contents, it is natural to expect scale and shift equivariance, _i.e._ normalization equivariance, from the denoising procedure emulated by \(f\). In image denoising, a majority of methods usually assume an additive white Gaussian noise model with variance \(\sigma^{2}\). The corruption model then reads \(y\sim\mathcal{N}(x,\sigma^{2}I_{n})\), where \(I_{n}\) denotes the identity matrix of size \(n\), and the noise standard deviation \(\sigma>0\) is generally passed as an additional argument to the denoiser ("non-blind" denoising). In this case, the augmented function \(f:(y,\sigma)\in\mathbb{R}^{n}\times\mathbb{R}^{+}_{*}\mapsto\mathbb{R}^{n}\) is said _normalization-equivariant_ if:

\[\forall(y,\sigma)\in\mathbb{R}^{n}\times\mathbb{R}^{+}_{*},\forall\lambda\in \mathbb{R}^{+}_{*},\forall\mu\in\mathbb{R},\ f(\lambda y+\mu,\lambda\sigma)= \lambda f(y,\sigma)+\mu\,, \tag{1}\]

as, according to the laws of statistics, \(\lambda y+\mu\sim\mathcal{N}(\lambda x+\mu,(\lambda\sigma)^{2}I_{n})\). In what follows, we give some well-known examples of traditional denoisers that are _normalization-equivariant_ (see proofs in Appendix D.2).

Noise-reduction filters:The most rudimentary methods for image denoising are the smoothing filters, among which we can mention the averaging filter or the Gaussian filter for the linear filters and the median filter which is nonlinear. These elementary "blind" denoisers all implement a _normalization-equivariant_ function. More generally, one can prove that a linear filter is _normalization-equivariant_ if and only if its coefficients add up to \(1\). In others words, _normalization-equivariant_ linear filters process images by affine combinations of pixels.

Patch-based denoising:The popular N(on)-L(ocal) M(eans) algorithm [20] and its variants [22, 23, 25] consist in computing, for each pixel, an average of its neighboring noisy pixels, weighted by the degree of similarity of patches they belong to. In other words, they processes images by convex combinations of pixels. More precisely, NLM can be defined as:

\[f_{\mathrm{NLM}}(y,\sigma)_{i}=\frac{1}{W_{i}}\sum_{y_{j}\in\Omega(y_{i})}e^{ -\frac{\|p(y_{i})-p(y_{j})\|_{2}^{2}}{h^{2}}}y_{j}\quad\text{with}\quad W_{i} =\sum_{y_{j}\in\Omega(y_{i})}e^{-\frac{\|p(y_{i})-p(y_{j})\|_{2}^{2}}{h^{2}}} \tag{2}\]

where \(y_{i}\) denotes the \(i^{th}\) component of vector \(y\), \(p(y_{i})\) represents the vectorized patch centered at \(y_{i}\), \(\Omega(y_{i})\) the set of its neighboring pixels and the smoothing parameter \(h\) is proportional to \(\sigma\) as proposed by several authors [21, 24, 25]. Defined as such, \(f_{\mathrm{NLM}}\) is a _normalization-equivariant_ function. More recently, NL-Ridge [27] and LIChI [28] propose to process images by linear combinations of similar patches and achieves state-of-the-art performance in unsupervised denoising. When restricting the coefficients of the combinations to sum to \(1\), that is imposing affine combination constraints, the resulting algorithms encode _normalization-equivariant_ functions as well.

TV denoising:Total variation (TV) denoising [18] is finally one of the most famous image denoising algorithm, appreciated for its edge-preserving properties. In its original form [18], a TV denoiser is defined as a function \(f:\mathbb{R}^{n}\times\mathbb{R}^{+}_{*}\mapsto\mathbb{R}^{n}\) that solves the following equality-constrained problem:

\[f_{\mathrm{TV}}(y,\sigma)=\operatorname*{arg\,min}_{x\in\mathbb{R}^{n}}\ \|x\|_{\mathrm{TV}}\quad\text{s.t.}\quad\|y-x\|_{2}^{2}=n\sigma^{2} \tag{3}\]

where \(\|x\|_{\mathrm{TV}}:=\|\nabla x\|_{2}\) is the total variation of \(x\in\mathbb{R}^{n}\). Defined as such, \(f_{\mathrm{TV}}\) is a _normalization-equivariant_ function.

### The case of neural networks

Deep learning hides a subtlety about normalization equivariance that deserves to be highlighted. Usually, the weights of neural networks are learned on a training set containing data all normalized to the same arbitrary interval \([a_{0},b_{0}]\). This training procedure improves the performance and allows for more stable optimization of the model. At inference, unseen data are processed within the interval \([a_{0},b_{0}]\) via a \(a\)-\(b\) linear normalization with \(a_{0}\leq a<b\leq b_{0}\) denoted \(\mathcal{T}_{a,b}\) and defined by:

\[\mathcal{T}_{a,b}:y\mapsto(b-a)\frac{y-\min(y)}{\max(y)-\min(y)}+a\,. \tag{4}\]

Note that this transform is actually the unique linear one with positive slope that exactly bounds the output to \([a,b]\). The data is then passed to the trained network and its response is finally returned to the original range via the inverse operator \(\mathcal{T}_{a,b}^{-1}\). This proven pipeline is actually relevant in light of the following proposition (see proof in Appendix D.1).
**Proposition 3**: \(\forall\,a<b\in\mathbb{R},\forall\,f:\mathbb{R}^{n}\mapsto\mathbb{R}^{m},\mathcal{T }_{a,b}^{-1}\circ f\circ\mathcal{T}_{a,b}\) _is a normalization-equivariant function._

While normalization-equivariance appears to be solved, a question is still remaining: how to choose the hyperparameters \(a\) and \(b\) for a given function \(f\)? Obviously, a natural choice for neural networks is to take the same parameters \(a\) and \(b\) as in the learning phase whatever the input image is, _i.e._\(a=a_{0}\) and \(b=b_{0}\), but are they really optimal? The answer to this question is generally negative. Figure 1 depicts an example of the phenomenon in image denoising, taken from a real-world application. In this example, the straightforward choice is largely sub-optimal. This suggests that there are always inherent performance leaks for deep neural networks due to the two degrees of freedom induced by the normalization (_i.e._, choice of \(a\) and choice of \(b\)). In addition, this poor conditioning can be a source of confusion and misinterpretation in critical applications.

### Categorizing image denoisers

Table 1 summarizes the equivariance properties of several popular denoisers, either conventional [18; 20; 27; 28; 19; 26; 29] or deep-learning-based [30; 32; 35; 36]. Interestingly, if _scale-equivariance_ is generally guaranteed for traditional denoisers, not all of them are equivariant to shifts. In particular, the widely used algorithms DCT [19] and BM3D [26] are sensitive to offsets, mainly because the hard thresholding function at their core is not _shift-equivariant_. Regarding the deep-learning-based networks, only DRUNet [32] is insensitive to scale because it is a bias-free convolutional neural network with only ReLU activation functions [1]. In the next section, we show how to adapt existing neural architectures to guarantee _normalization-equivariance_ without loss of performance and study the resulting class of parameterized functions \((f_{\theta})\).

## 4 Design of Normalization-Equivariant Networks

### Affine convolutions

To justify the introduction of a new type of convolutional layers, let us study one of the most basic neural network, namely the linear (parameterized) function \(f_{\Theta}:x\in\mathbb{R}^{n}\mapsto\Theta x\), where parameters \(\Theta\) are a matrix of \(\mathbb{R}^{m\times n}\). Indeed, \(f_{\Theta}\) can be interpreted as a dense neural network with no bias, no hidden layer and no activation function. Obviously, \(f_{\Theta}\) is always _scale-equivariant_, whatever the weights \(\Theta\). As for the _shift-equivariance_, a simple calculation shows

\begin{table}
\begin{tabular}{l c c c c c c c|c c c c}  & TV & NLM & NL-Ridge & LIChI & DCT & BM3D & WNNNM & DnCNN & NLRN & SwinIR & DRUNet \\ \hline Scale & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✗ & ✗ & ✗ & ✓ \\ Shift & ✓ & ✓ & ✓ & ✓ & ✗ & ✗ & ✗ & ✗ & ✗ & ✗ & ✗ \\ \hline \end{tabular}
\end{table}
Table 1: Equivariance properties of several image denoisers (left: traditional, right: learning-based)

Figure 1: Influence of normalization for deep-learning-based image denoising. The raw input data is a publicly available real noisy image of the _Convallaria_ dataset [17]. “Blind” DnCNN [30] with official pre-trained weights is used for denoising and is applied on four different normalization intervals displayed in red, each of which being included in \([0,1]\) over which it was learned. PSNR is calculated with the average of \(100\) independent noisy static acquisitions of the same sample (called ground truth). Interestingly, the straightforward interval \([0,1]\) does not give the best results. Normalization intervals are (a) \([0,1]\), (b) \([0.08,0.12]\), (c) \([0.48,0,52]\) and (d) \([0.64,0.96]\). In the light of the denoising results \((b)\)-\((c)\) and \((c)\)-\((d)\), DnCNN is neither _shift-equivariant_, nor _scale-equivariant_.
that:

\[x\mapsto\Theta x\text{ is {shift-equivariant} }\;\Leftrightarrow\;\forall x\in\mathbb{R}^{n},\forall\mu\in \mathbb{R},\Theta(x+\mu\mathbf{1}_{n})=\Theta x+\mu\mathbf{1}_{m}\; \Leftrightarrow\;\Theta\mathbf{1}_{n}=\mathbf{1}_{m}\,. \tag{5}\]

Therefore, \(f_{\Theta}\) is _normalization-equivariant_ if and only if each row of matrix \(\Theta\) sums to \(1\). In other words, for the _normalization-equivariance_ to hold, the rows of \(\Theta\) must encode weights of affine combinations. Transposing the demonstration to any convolutional neural network, a convolutional layer preserves the _normalization-equivariance_ if and only if the weights of the each convolutional kernel sums to \(1\). In the following, we call such convolutional layers "affine convolutions".

As a consequence, since _normalization-equivariance_ is preserved through function composition, concatenation and affine combination (see Prop. 2), a (linear) convolutional neural network composed of only affine convolutions with no bias and possibly skip or _affine_ residual connections (trainable affine combination of two layers), is guaranteed to be _normalization-equivariant_, provided that padding is performed with existing features (reflect, replicate or circular padding for example). Obviously, in their current state, these neural networks are of little interest, as linear functions do not encode best-performing functions for many applications, image denoising being no exception. Nevertheless, based on such networks, we show in the next subsection how to introduce nonlinearities without breaking the _normalization-equivariance_.

### Channel-wise sort pooling as a normalization-equivariant alternative to ReLU

The first idea that comes to mind is to apply a nonlinear activation function \(\varphi:\mathbb{R}\mapsto\mathbb{R}\) preserving _normalization-equivariance_ after each affine convolution. In other words, we look for a nonlinear solution \(\varphi\) of the characteristic functional equation of _normalization-equivariant_ functions (see Def. 1) for \(n=1\). Unfortunately, the unique solution is the identity function which is linear (see Prop. 1: \(\mathcal{S}\cap\operatorname{Span}(\mathbf{1}_{n})^{\perp}=\emptyset\) for \(n=1\)). Therefore, activation functions that apply element-wise are to be excluded.

To find interesting nonlinear functions, one needs to examine multi-dimensional activation functions, _i.e._ ones of the form \(\varphi:\mathbb{R}^{n}\mapsto\mathbb{R}^{m}\) with \(n\geq 2\). In order to preserve the dimensions of the neural layers and to limit the computational costs, we focus on the case \(n=m=2\), meaning that \(\varphi\) processes pre-activated neurons by pairs. According to Prop. 1, _normalization-equivariant_ functions are completely determined by their values on \(\mathcal{S}\cap\operatorname{Span}(\mathbf{1}_{n})^{\perp}\), which reduces to the characteristic set \(\mathcal{C}=\{-u,u\}\), where \(u=(-1/\sqrt{2},1/\sqrt{2})\), when considering the Euclidean distance of \(\mathbb{R}^{2}\). By arbitrarily setting \(\varphi(-u)=\varphi(u)=u\), the resulting function simply reads:

\[\varphi:(x,y)\in\mathbb{R}^{2}\mapsto\begin{pmatrix}\min(x,y)\\ \max(x,y)\end{pmatrix}\,, \tag{6}\]

which is nothing else that the sorting function in \(\mathbb{R}^{2}\). More generally, it is easy to show that all the sorting functions of \(\mathbb{R}^{n}\) are _normalization-equivariant_. The good news is that these functions are nonlinear as soon as \(n\geq 2\). Therefore, they are candidates to replace the conventional activation functions such as the popular ReLU (rectified linear unit) function.

Since the sorting function (6) is to be applied on non-overlapping pairs of neurons, the partitioning of layers needs to be determined. In order not to mix unrelated neurons, we propose to apply this two-dimensional activation function

Figure 2: Illustration of the proposed alternative for replacing the traditional scheme “convolution + element-wise activation function” in convolutional neural networks: affine convolutions supersede ordinary ones by restricting the coefficients of each kernel to sum to one and the proposed sort pooling patterns introduce nonlinearities by sorting two by two the pre-activated neurons along the channels.
channel-wisely across layers and call this operation "sort pooling" in reference to the max pooling operation, widely used for downsampling, and from which it can be effectively implemented. Figure 2 illustrates the sequence of the two proposed innovations, namely affine convolution followed by channel-wise sort pooling, to replace the traditional scheme "conv+ReLU", while guaranteeing _normalization-equivariance_.

### Encoding adaptive affine filters

Based on Prop. 2, we can formulate the following proposition which tells more about the class of parameterized functions \((f_{\theta})\) encoded by the proposed networks (see proof in Appendix D.1).

**Proposition 4**: _Let \(f_{\theta}^{\text{NE}}:\mathbb{R}^{n}\mapsto\mathbb{R}^{m}\) be a CNN composed of only:_

* _affine convolution kernels with no bias and where padding is made of existing features,_
* _sort pooling nonlinearities,_
* _possibly skip or affine residual connections._

_Then, \(f_{\theta}^{\text{NE}}\) is a normalization-equivariant continuous piecewise-linear function with finitely many pieces. Moreover, on each piece represented by the vector \(y_{r}\),_

\[f_{\theta}^{\text{NE}}(y)=A_{\theta}^{y_{r}}y,\text{ with }A_{\theta}^{y_{r}} \in\mathbb{R}^{m\times n}\text{ such that }A_{\theta}^{y_{r}}\mathbf{1}_{n}=\mathbf{1}_{m}\,.\]

In Prop. 4, the subscripts on \(A_{\theta}^{y_{r}}\) serve as a reminder that this matrix depends on the sort pooling activation patterns, which in turn depend on both the input vector \(y\) and the weights \(\theta\). As already revealed for bias-free networks with ReLU [1], \(A_{\theta}^{y_{r}}\) is the Jacobian matrix of \(f_{\theta}^{\text{NE}}\) taken at any point \(y\) in the interior of the piece represented by vector \(y_{r}\). Moreover, as \(A_{\theta}^{y_{r}}\mathbf{1}_{n}=\mathbf{1}_{m}\), the output vector of such networks are locally made of fixed affine combinations of the entries of the input vector. And since a CNN has a limited receptive field centered on each pixel, \(f_{\theta}^{\text{NE}}\) can be thought of as an adaptive filter that produces an estimate of each pixel through a custom affine combination of pixels. By examining these filters in the case of image denoising (see Fig. 3), it becomes apparent that they vary in their characteristics and are intricately linked to the contents of the underlying images. Indeed, these filters are specifically designed to cater to the specific local features of the noisy image: averaging is done over uniform areas without affecting the sharpness of edges. Note that this behavior has already been extensively studied by [1] for unconstrained filters.

The total number of fixed adaptive affine filters depends on the weights \(\theta\) of the network \(f_{\theta}^{\text{NE}}\) and is bounded by \(2^{S}\) where \(S\) represents the total number of sort pooling patterns traversed to get from the receptive filed to its final pixel. Obviously, this upper bound grows exponentially with \(S\), suggesting that a limited number of sort pooling operations

Figure 3: Visual comparisons of the generalization capabilities of a _scale-equivariant_ neural network (left) and its _normalization-equivariant_ counterpart (right) for Gaussian noise. Both networks were trained for Gaussian noise at noise level \(\sigma=25\) exclusively. The adaptive filters (rows of \(A_{\theta}^{y_{r}}\) in Prop. 4) are indicated for two particular pixels as well as the sum of their coefficients (note that some weights are negative, indicated in red). The _scale-equivariant_ network tends to excessively smooth out the image when evaluated at a lower noise level, whereas the _normalization-equivariant_ network is more adaptable and considers the underlying texture to a greater extent.
may generate an extremely large number of filters. Interestingly, if ReLU activation functions where used instead, the upper bound would reach \(2^{2S}\).

## 5 Experimental results

We demonstrate the effectiveness and versatility of the proposed methodology in the case of image denoising. To this end, we modify two well-established neural network architectures for image denoising, chosen for both their simplicity and efficiency, namely DRUNet [32]: a state-of-the-art U-Net with residual connections [2]; and FDnCNN, the unpublished flexible variant of the popular DnCNN [30]: a simple feedforward CNN that chains "conv+ReLU" layers with no downsampling, no residual connections and no batch normalization during training [3], and with a tunable noise level map as additional input [31]. We show that adapting these networks to become _normalization-equivariant_ does not adversely affect performance and, better yet, increases their generalization capabilities. For each scenario, we train three variants of the original Gaussian denoising network for grayscale images: _ordinary_ (original network with additive bias), _scale-equivariant_ (bias-free variation with ReLU [1]) and our _normalization-equivariant_ architecture (see Fig. 2). Details about training and implementations can be found in Appendix A and B; the code is available at [https://github.com/sherbret/normalization_equivariant_nn/](https://github.com/sherbret/normalization_equivariant_nn/). Unless otherwise noted, all results presented in this paper are obtained with DRUNet [32]; similar outcomes can be achieved with FDnCNN [30] architecture (see Appendix C).

Finally, note that both DRUNet [32] and FDnCNN [30] can be trained as "blind" but also as "non-blind" denoisers and thus achieve increased performance, by passing an additional noisemap as input. In the case of additive white Gaussian noise of variance \(\sigma^{2}\), the noisemap is constant equal to \(\sigma\mathbf{1}_{n}\) and the resulting parameterized functions can then be put mathematically under the form \(f_{\theta}:(y,\sigma)\in\mathbb{R}^{n}\times\mathbb{R}^{+}_{*}\mapsto\mathbb{ R}^{n}\). In order to integrate this feature to _normalization-equivariant_ networks as well, a slight modification of the first affine convolutional layer must be made. Indeed, by adapting the proof (5) to the case (1), we can show that the first convolutional layer must be affine with respect to the input image \(y\) only - the coefficients of the kernels acting on the image pixels add up to \(1\) - while the other coefficients of the kernels need not be constrained.

### The proposed architectural modifications do not degrade performance

The performance, assessed in terms of PSNR values, of our _normalization-equivariant_ alternative (see Fig. 2) and of its _scale-equivariant_ and _ordinary_ counterparts is compared in Table 2 for "non-blind" architectures on two popular datasets [37]. We can notice that the performance gap between two different variants is less than 0.05 dB at most for all noise levels, which is not significant. This result suggests that the class of parameterized functions \((f_{\theta})\) currently used in image denoising can drastically be reduced at no cost. Moreover, it shows that it is possible to dispense with activation functions, such as the popular ReLU: nonlinearities can simply brought by sort pooling patterns. In terms of subjective visual evaluation, we can draw the same conclusion since images produced by two architectural variants inside the training range are hardly distinguishable (see Fig. 3 at \(\sigma=25\)).

### Increased robustness across noise levels

S. Mohan _et al._[1] revealed that bias-free neural networks with ReLU, which are _scale-equivariant_, could much better generalize when evaluated at new noise levels beyond their training range, than their counterparts with bias that systematically overfit. Even if they do not fully elucidate how such networks achieve this remarkable generalization, they suggest that _scale-equivariance_ certainly plays a major role. What about _normalization-equivariance_ then? We

\begin{table}
\begin{tabular}{c c c c c} \hline \hline Dataset & \multicolumn{3}{c}{Set12} & BSD68 \\ \hline \hline \multicolumn{2}{c}{Noise level \(\sigma\)} & 15 / 25 / 50 & 15 / 25 / 50 \\ \hline \multirow{3}{*}{DRUNet [32]} & _ordinary_ & 33.23 / 30.92 / 27.87 & 31.89 / 29.44 / 26.54 \\  & _scale-equiviv_ & 33.25 / 30.94 / 27.90 & 31.91 / 29.48 / 26.59 \\  & _norm-equiviv_ & 33.20 / 30.90 / 27.85 & 31.88 / 29.45 / 26.55 \\  & _ordinary_ & 32.87 / 30.49 / 27.28 & 31.69 / 29.22 / 26.27 \\ FDnCNN [30] & _scale-equiviv_ & 32.85 / 30.49 / 27.29 & 31.67 / 29.20 / 26.25 \\  & _norm-equiviv_ & 32.85 / 30.50 / 27.27 & 31.69 / 29.22 / 26.25 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The PSNR (dB) results of “non-blind” deep-learning-based methods applied to popular grayscale datasets corrupted by synthetic white Gaussian noise with \(\sigma=15\), \(25\) and \(50\).
have compared the robustness faculties of the three variants of networks when trained at a fixed noise level \(\sigma\) for Gaussian noise. Figure 4 summarizes the explicit results obtained: _normalization-equivariance_ pushes generalization capabilities of neural networks one step further. While performance is identical to their _scale-equivariant_ counterparts when evaluated at higher noise levels, the _normalization-equivariant_ networks are, however, much more robust at lower noise levels. This phenomenon is also illustrated in Fig. 3.

Demystifying robustnessLet \(x\) be a clean patch of size \(n\), representative of the training set on which a CNN \(f_{\theta}\) was optimized to denoise its noisy realizations \(y=x+\varepsilon\) with \(\varepsilon\sim\mathcal{N}(0,\sigma^{2}I_{n})\) (denoising at a fixed noise level \(\sigma\) exclusively). Formally, we note \(x\in\mathcal{D}\subset\mathbb{R}^{n}\), where \(\mathcal{D}\) is the space of representative clean patches of size \(n\) on which \(f_{\theta}\) was trained. We are interested in the output of \(f_{\theta}\) when it is evaluated at \(x+\lambda\varepsilon\) (denoising at noise level \(\lambda\sigma\)) with \(\lambda>0\). Assuming that \(f_{\theta}\) encodes a _normalization-equivariant_ function, we have:

\[\forall\lambda\in\mathbb{R}_{*}^{+},\forall\mu\in\mathbb{R},\;f_{\theta}(x+ \lambda\varepsilon)=\lambda f_{\theta}((x-\mu)/\lambda+\varepsilon)+\mu\,. \tag{7}\]

Figure 4: Comparison of the performance of our _normalization-equivariant_ alternative with its _scale-equivariant_ and _ordinary_ counterparts for Gaussian denoising with the same architecture on Set12 dataset. The vertical blue line indicates the unique noise level on which the networks were trained exclusively (from left to right: \(\sigma=50\), \(\sigma=25\) and \(\sigma=10\)). In all cases, _normalization-equivariant_ networks generalize much more robustly beyond the training noise level.

Figure 5: Denoising results for example images of the form \(y=x+\lambda\varepsilon\) (see notations of subsection 5.2) with \(\sigma=25/255\) and \(x\in[0,1]^{n}\), by CNNs specialized for noise level \(\sigma\) only. \(f_{\theta}^{\varnothing}\), \(f_{\theta}^{\text{SE}}\) and \(f_{\theta}^{\text{NE}}\) denote the _ordinary_, _scale-equivariant_ and _normalization-equivariant_ variants, respectively. In order to get the best results with \(f_{\theta}^{\varnothing}\) and \(f_{\theta}^{\text{SE}}\), it is necessary know the renormalization parameters \((\lambda,\mu)\) such that \((x-\mu)/\lambda\) belongs to \(\mathcal{D}\subset[0,1]^{n}\) (see subsection 5.2). Note that for \(f_{\theta}^{\text{SE}}\), it is however sufficient to know only \(\mu\) as \(\lambda\) is implicit by construction. In contrast, \(f_{\theta}^{\text{NE}}\) can be applied directly.
The above equality shows how such networks can deal with noise levels \(\lambda\sigma\) different from \(\sigma\): _normalization-equivariance_ simply brings the problem back to the denoising of an implicitly renormalized image patch with fixed noise level \(\sigma\). Note that this artificial change of noise level does not make this problem any easier to solve as the signal-to-noise ratio is preserved by normalization. Obviously, the denoising result of \(x+\lambda\varepsilon\) will be all the more accurate as \((x-\mu)/\lambda\) is a representative patch of the training set. In other words, if \((x-\mu)/\lambda\) can still be considered to be in \(\mathcal{D}\), then \(f_{\theta}\) should output a consistent denoised image patch. For a majority of methods [30, 32, 31], training is performed within the interval \([0,1]\) and therefore \(x/\lambda\) still belongs generally to \(\mathcal{D}\) for \(1<\lambda<10\) (contraction), but this is much less true for \(\lambda<1\) (stretching) for the reason that it may exceed the bounds of the interval \([0,1]\). This explains why _scale-equivariant_ functions do not generalize well to noise levels lower than their training one. In contrast, _normalization-equivariant_ functions can benefit from the implicit extra adjustment parameter \(\mu\). Indeed, there exists some cases where the stretched patch \(x/\lambda\) is not in \(\mathcal{D}\) but \((x-\mu)/\lambda\) is (see Fig. 4(b)). This is why _normalization-equivariant_ networks are more able to generalize at low noise levels. Note that, based on this argument, _ordinary_ neural networks trained at a fixed noise level \(\sigma\) can also be used to denoise images at noise level \(\lambda\sigma\), provided that a correct normalization is done beforehand. However, this time the normalization is explicit: the exact scale factor \(\lambda\), and possibly the shift \(\mu\), must be known (see Fig. 4(a)).

## 6 Conclusion and perspectives

In this work, we presented an original approach to adapt the architecture of existing neural networks so that they become _normalization-equivariant_, a property highly desirable and expected in many applications such that image denoising. We argue that the classical pattern "conv+ReLU" can be favorably replaced by the two proposed innovations: affine convolutions that ensure that all coefficients of the convolutional kernels sum to one; and channel-wise sort pooling nonlinearities as a substitute for all activation functions that apply element-wise, including ReLU or sigmoid functions. Despite these two important architectural changes, we show that the performance of these alternative networks is not affected in any way. On the contrary, thanks to their better-conditioning, they benefit, in the context of image denoising, from an increased interpretability and especially robustness to variable noise levels both in practice and in theory.

More generally, the proposed channel-wise sort pooling nonlinearities may potentially change the way we commonly understand neural networks: the usual paradigm that neurons are either active ("fired") or inactive, is indeed somewhat shaken. With sort pooling nonlinearities, neurons are no longer static but they "wiggle and mingle" according to the received signal. We believe that this discovery may help building new neural architectures, potentially with stronger theoretical guarantees, and more broadly, may also open the doors for novel perspectives in deep learning.

## Acknowledgments

This work was supported by Bpfrance agency (funding) through the LiChIE contract. Computations were performed on the Inria Rennes computing grid facilities partly funded by France-BioImaging infrastructure (French National Research Agency - ANR-10-INBS-04-07, "Investments for the future"). We would like to thank R. Fraisse (Airbus) for fruitful discussions.
