# Guidelines for the Regularization of Gammas in Batch Normalization for Deep
Residual Networks

Bum Jun Kim
POSTECH
<EMAIL_ADDRESS>
&Hyeyeon Choi
POSTECH
<EMAIL_ADDRESS>
Hyeonah Jang
POSTECH
<EMAIL_ADDRESS>
&Dong Gu Lee
POSTECH
<EMAIL_ADDRESS>
&Wonseok Jeong
POSTECH
<EMAIL_ADDRESS>
&Sang Woo Kim
POSTECH
<EMAIL_ADDRESS>

###### Abstract

$L_{2}$ regularization for weights in neural networks is widely used as a
standard training trick. However, $L_{2}$ regularization for $\gamma$, a
trainable parameter of batch normalization, remains an undiscussed mystery and
is applied in different ways depending on the library and practitioner. In
this paper, we study whether $L_{2}$ regularization for $\gamma$ is valid. To
explore this issue, we consider two approaches: 1) variance control to make
the residual network behave like identity mapping and 2) stable optimization
through the improvement of effective learning rate. Through two analyses, we
specify the desirable and undesirable $\gamma$ to apply $L_{2}$ regularization
and propose four guidelines for managing them. In several experiments, we
observed the increase and decrease in performance caused by applying $L_{2}$
regularization to $\gamma$ of four categories, which is consistent with our
four guidelines. Our proposed guidelines were validated through various tasks
and architectures, including variants of residual networks and transformers.

##  1 Introduction

Deep neural networks have exhibited remarkable performance in various fields.
Previously, large neural networks with deep and wide architectures were
considered difficult to train. However, various regularization techniques such
as $L_{2}$ regularization [26], batch normalization (BN) [12], and residual
learning [8] have made it possible to train large neural networks, leading to
successful performance.

Since the classic machine learning era, $L_{2}$ regularization has been
applied as a restriction on the weights $W$ of neural networks and has not
been applied to bias $b$. However, trainable parameters in modern neural
networks are not limited to weights $W$ and bias $b$. BN outputs
$\gamma_{i}\hat{x_{i}}+\beta_{i}$ from normalized feature $\hat{x}$, where
$\gamma$ and $\beta$ are also trainable parameters. Because $\beta$ plays a
similar role to bias $b$, we ignore $L_{2}$ regularization of $\beta$.
However, $\gamma$ controls the scale of the feature map and can be viewed as a
special case of weights (Section 3). Despite the similar roles of $\gamma$ and
weights, $L_{2}$ regularization of the $\gamma$ parameters of BN remains an
undiscussed mystery. Moreover, each deep learning library and practice
provides a different method:

#### TensorFlow and Keras

To implement $L_{2}$ regularization, a regularizer option needs to be set for
each layer. For example, kernel_regularizer can be applied to the convolution
and fully connected layer to implement $L_{2}$ regularization on weight $W$.
$L_{2}$ regularization of $\gamma$ is functionally supported through
gamma_regularizer, which may be set to the BN layer. In practice, however,
gamma_regularizer is rarely used. To our knowledge, in the TensorFlow and
Keras official tutorials and code, practical use of gamma_regularizer does not
exist. In other words, $L_{2}$ regularization in TensorFlow and Keras
generally means using kernel_regularizer on weights, and $L_{2}$
regularization has not been applied to $\gamma$.

#### PyTorch

$L_{2}$ regularization is implemented by applying a weight decay at the
optimizer level. However, the weight decay of PyTorch is applied to all
trainable parameters, including all $W$, $b$, $\gamma$, and $\beta$. Although
the validity of weight decay for $b$ and $\beta$ is also arguable, we focus on
whether PyTorch’s practice of applying weight decay to $\gamma$ is valid.

#### Practice and empirical observation

In several work [10, 13, 6], $L_{2}$ regularization was not applied $\gamma$
and $\beta$. However, [22, 24] used $L_{2}$ regularization on $\gamma$ and
$\beta$, while [22] turned off it during fine-tuning. Reference [18]
empirically observed that $L_{2}$ regularization on $\gamma$ and $\beta$ often
improves performance depending on the architecture.

Table 1: Preview of our guidelines on $L_{2}$ regularization. | TensorFlow | PyTorch | Our Guidelines
---|---|---|---
$W$ | Applied ✓ | Applied ✓ | Applicable ✓
$\gamma_{last}$ | Rarely applied ✗ | Applied ✓ | Applicable ✓
$\gamma_{down}$ | Rarely applied ✗ | Applied ✓ | Inapplicable ✗
$\gamma_{0}$ | Rarely applied ✗ | Applied ✓ | Inapplicable ✗
$\gamma_{others}$ | Rarely applied ✗ | Applied ✓ | Applicable ✓

Because these practices lack theoretical analysis, in this paper, we explore
whether it is desirable to apply $L_{2}$ regularization to the $\gamma$
parameter of BN. First, we claim that $\gamma$ plays a role in controlling
variance in residual networks. We show that the variance of the features in
the residual block is either accumulated or reset according to the layer
arrangement. For better behavior in residual networks, we propose a strategy
of making the accumulated variance small and the reset variance large (Section
2). Second, we present an analysis of the effective learning rate from
optimization perspective of $\gamma$ (Section 3). Through our theoretical
analysis, we present four guidelines for managing $\gamma$ (Table 1).

The validity of the four guidelines is confirmed through several experiments
(Section 4). We demonstrate a performance decrease due to incorrect $L_{2}$
regularization on $\gamma$ and a performance increase due to correct $L_{2}$
regularization on $\gamma$. We observed this phenomenon in various residual
networks and transformers.

##  2 Variance analysis in residual networks

In this paper, we target residual networks because they are widely used as a
standard architecture and involve many variants. In residual networks, an
input image is first passed through the early stage. Then, four stages are
applied, and each stage is composed of several residual blocks. Each residual
block consists of a skip connection and a residual branch, in which several
weight, BN, and ReLU layers are placed in the residual branch.

Our assumption is that the condition when the skip connection dominates over
the residual branch, which makes the residual block behave more like an
identity mapping, is advantageous for residual network training. In fact, in
the original ResNet paper, residual learning was introduced with the intention
of modelling identity mappings [8]. Reference [6] reported that initializing
$\gamma=0$ in the last BN causes the residual block to behave like an identity
function and eases optimization. Reference [3] found that a residual block
behaves like an identity mapping because the variance of the skip connection
is larger than the variance of the residual branch. In their paper, the
$\gamma$ parameter of BN was assumed to be 1, but we show here that variance
in the residual block varies according to $\gamma$.

We consider the bias parameter $b$ in the weight layers and $\beta$ in the BN
layers to be 0. $L_{2}$ regularization of $b$ or $\beta$ is ignored in this
paper. In BN, we assume that the mini-batch size is sufficiently large, and
each feature map is normalized to $\mathcal{N}(0,1)$ and then rescaled using
$\gamma$. In other words, the $i$-th element of BN with $\gamma_{i}$ outputs
$B_{i}(x)$, where $E[B_{i}(x)]=0$ and $Var[B_{i}(x)]=\gamma_{i}^{2}$. The term
"weight layers" used below indicates convolution layers in ResNet, but will be
described by generalizing to fully connected layers. We assume that each
element of the weights comes from He initialization
$\mathcal{N}(0,2/fan_{in})$ [7].

Figure 1: Basic block in (left) PreActResNet and (right) original ResNet
(Cases 1 and 2).

#### Case 1. Basic block in PreActResNet

PreActResNet [9] is a modified version of the original ResNet [8], and is also
called ResNetV2. The residual branch of PreActResNet consists of two or three
[BN–ReLU–Weight] blocks (Figure 1, Left). The $l$-th residual block outputs
$x_{l+1}$, which is the sum of the residual branch $f_{l}(x_{l})$ and skip
connection $x_{l}$, i.e., $x_{l+1}=x_{l}+f_{l}(x_{l})$. Because the covariance
between the residual branch and the skip connection is zero [3], we have
$Var[x_{l+1}]=Var[x_{l}]+Var[f_{l}(x_{l})]$.

Here, we examine the variance of the residual branch $Var[f_{l}(x_{l})]$.
Consider the residual branch with two [BN–ReLU–Weight] blocks. Because of the
normalization in the second BN, the $\gamma$ of the first BN and $\left\lVert
W_{1}\right\rVert$ do not affect the variance of the residual branch.
Therefore, the variance of the residual branch is determined by the second
[BN–ReLU–Weight] block. By the mean and variance of $W$ from He initialization
and BN, the variance of the $i$-th element of the residual branch is

$\displaystyle Var[f_{l,i}(x_{l})]$
$\displaystyle=\sum_{j}^{fan_{in}}Var[W_{2,ij,l}]\cdot
E[ReLU(B_{2,l,j}(x_{l}))^{2}]$ (1) $\displaystyle=2\cdot
E[ReLU(B_{2,l,i}(x_{l}))^{2}]$ (2)
$\displaystyle=E[B_{2,l,i}(x_{l})^{2}]=Var[B_{2,l,i}(x_{l})]$ (3)
$\displaystyle=\gamma_{2,l,i}^{2}.$ (4)

The third equality holds because $E[ReLU(X)^{2}]=\frac{1}{2}E[X^{2}]$ for
normalized $X$ [7]. Thus,

$\displaystyle Var[x_{l+1,i}]=Var[x_{l,i}]+\gamma_{2,l,i}^{2}.$ (5)

Starting from the $s$-th residual block, imagine that the variance of the
residual branch is accumulated in a row. The variance of the $l$-th residual
block is

$\displaystyle Var[x_{l,i}]=Var[x_{s,i}]+\sum_{m=s}^{l-1}\gamma_{2,m,i}^{2},$
(6)

where $x_{s}$ denotes the first input feature map for each stage of ResNet. As
such, in the residual block, the variance of the residual branch is
accumulated. Here, we want to make the residual block satisfy
$Var[x_{l,i}]>Var[f_{l,i}(x_{l})]$ so that the skip connection dominates over
the residual branch. From Eq. 4 and Eq. 6, the inequality
$Var[x_{s,i}]+\sum_{m=s}^{l-1}\gamma_{2,m,i}^{2}>\gamma_{2,l,i}^{2}$ requires
two conditions:

  1. 1.

$\gamma_{2,l,i}^{2}$ should be small for all $l$;

  2. 2.

$Var[x_{s,i}]$ should be large.

The variance at the stage starting point in the second condition is discussed
later in Cases 3 and 4. According to the first condition, $\gamma$ in the
second BN in all residual blocks should be small to reduce the variance of the
residual branch. Similarly, when the residual branch consists of three
[BN–ReLU–Weight] blocks, the $\gamma$ in the third BN should be small. Thus,
in order for the skip connection to dominate over the residual branch, we
should control $\gamma$ in the last BN in the residual branch. To ensure that
those called $\gamma_{last}$ are small during training, we recommend applying
$L_{2}$ regularization on $\gamma_{last}$. In summary, we present the
following guideline:

#### Guideline 1.

Because residual block plays the role of variance accumulation, we should
decay $\gamma_{last}$.

#### Case 2. Basic block in the original ResNet

We check whether Guideline 1 is also applicable to the original ResNet. The
original ResNet, also called ResNetV1, constitutes a residual branch with two
or three [Weight–BN–ReLU] blocks, but the last ReLU is deployed after addition
(Figure 1, Right). First, we examine the variance of residual branch
$Var[f_{l}(x_{l})]$. Because the residual branch ends with BN, the variance of
the residual branch is equal to the variance of the last BN, i.e.,
$Var[f_{l,i}(x_{l})]=\gamma_{last,l,i}^{2}$.

Contrary to PreActResNet, here, because ReLU is applied after addition,
$x_{l+1}=ReLU(x_{l}+f_{l}(x_{l}))$. This ReLU after addition makes a different
variance accumulation. Ignoring $E[x_{l,i}]$, we have

$\displaystyle Var[x_{l+1,i}]$
$\displaystyle=\frac{1}{2}E[(x_{l,i}+f_{l,i}(x_{l}))^{2}]$ (7)
$\displaystyle=\frac{1}{2}(E[x_{l,i}^{2}]+E[f_{l,i}(x_{l})^{2}])$ (8)
$\displaystyle=\frac{1}{2}(Var[x_{l,i}]+Var[f_{l,i}(x_{l})])$ (9)
$\displaystyle=\frac{1}{2}(Var[x_{l,i}]+\gamma_{last,l,i}^{2}).$ (10)

Thus, the added variance is halved. Here, we introduce the following
substitutions: $a_{l}=2^{l}Var[x_{l,i}]$ and
$b_{l}=2^{l}\gamma_{last,l,i}^{2}$. Rewriting Eq. 10, we have
$a_{l+1}=a_{l}+b_{l}$, and thus $a_{l}=a_{s}+\sum_{m=s}^{l-1}b_{m}$.
Therefore, we obtain

$\displaystyle Var[x_{l,i}]$
$\displaystyle=\frac{1}{2^{l-s}}Var[x_{s,i}]+\sum_{m=s}^{l-1}\frac{1}{2^{l-m}}\gamma_{last,m,i}^{2}.$
(11)

As such, the variance of the residual branch is accumulated here as well, but
because the variance of the last ReLU is halved, the variance of the previous
block decays. We call this _half variance accumulation_. For the skip
connection to dominate, the condition $Var[x_{l,i}]>Var[f_{l,i}(x_{l})]$
requires that 1) $\gamma_{last,l,i}^{2}$ should be small and 2) $Var[x_{s,i}]$
should be large. Therefore, it is desirable to decay $\gamma_{last}$, and
thus, Guideline 1 is valid for the original ResNet as well.

Figure 2: Downsampling block in (left) PreActResNet and (right) original
ResNet (Case 3).

#### Case 3. Downsampling block in ResNet

Convolutional neural networks downsample high-dimensional image features to
low-dimensional ones using pooling layers [14]. For ResNet, downsampling is
performed using strided convolution [17] as well as pooling. For the original
ResNet, at the end of each stage, a downsampling block (Figure 2, Right) is
applied, where the strided convolution is used in both the first weight layers
in the residual branch and the skip path. Here, the skip path of the
downsampling block is composed of strided convolution and BN rather than
identity. We denote the skip path output as
$g_{l}(x_{l})=B_{down,l}(W_{down,l}x_{l})$. Because each branch ends with BN,
the variance of each branch is determined by the last BN as follows:

$\displaystyle Var[g_{l,i}(x_{l})]$ $\displaystyle=\gamma_{down,l,i}^{2},$
(12) $\displaystyle Var[f_{l,i}(x_{l})]$
$\displaystyle=\gamma_{last,l,i}^{2}.$ (13)

From $x_{l+1}=ReLU(g_{l}(x_{l})+f_{l}(x_{l}))$, the variance after the
downsampling block is

$\displaystyle Var[x_{l+1,i}]$
$\displaystyle=\frac{1}{2}(Var[g_{l,i}(x_{l})]+Var[f_{l,i}(x_{l})])$ (14)
$\displaystyle=\frac{1}{2}(\gamma_{down,l,i}^{2}+\gamma_{last,l,i}^{2}).$ (15)

Note that the output variance of the downsampling block is not affected by the
input variance and is newly determined by $\gamma_{down}$ and $\gamma_{last}$.
We call this _variance reset_.

Again, it is desirable to decay $\gamma_{last}$ so that the residual branch
does not dominate. However, it is not desirable to decay $\gamma_{down}$.
There are two reasons for this. First, in order for the skip path in the
downsampling block to dominate, we should decay $\gamma_{last}$ and should not
decay $\gamma_{down}$. Second, it is desirable to have a large reset variance.
The output variance of the downsampling block becomes the input variance of
the successive $(l+1)$-th residual block, which is the variance at the stage
starting point. In Cases 1 and 2, we confirmed that a large $Var[x_{s,i}]$ is
favorable. Thus, a large output variance of the downsampling block is
desirable. In other words, if we decay both $\gamma_{down}$ and
$\gamma_{last}$, the reset variance decreases, which is undesirable.
Therefore, we claim that $\gamma_{down}$ should not be subjected to $L_{2}$
regularization to ensure that the reset variance appears dominant afterward.
In this regard, we present the following guideline:

#### Guideline 2.

Because the downsampling block plays the role of variance reset, we should not
decay $\gamma_{down}$.

Figure 3: Performance degradation caused by $L_{2}$ regularization on
$\gamma_{down}$ with various $\lambda$ for ResNet-152 on the PET dataset.

Now, we check whether Guideline 2 is also applicable to the downsampling block
of PreActResNet. In the downsampling block of PreActResNet, before starting
both the residual branch and skip path, block input $x_{l}$ is subjected to
the first BN and ReLU (Figure 2, Left). The first BN does not affect the
variance of the residual branch but does affect the variance of the skip path.
For skip path $g_{l}(x_{l})=W_{down,l}ReLU(B_{1,l}(x_{l}))$, we have
$Var[g_{l,i}(x_{l})]=2\cdot E[ReLU(B_{1,l,i}(x_{l}))^{2}]=\gamma_{1,l,i}^{2}$.
For the residual branch, similar to Case 1, we have
$Var[f_{l,i}(x_{l})]=\gamma_{2,l,i}^{2}$. Thus, the variance after residual
block is

$\displaystyle Var[x_{l+1,i}]$
$\displaystyle=Var[g_{l,i}(x_{l})]+Var[f_{l,i}(x_{l})]$ (16)
$\displaystyle=\gamma_{1,l,i}^{2}+\gamma_{2,l,i}^{2}.$ (17)

Thus, the downsampling block of PreActResNet plays the role of variance reset
as well. To ensure that the reset variance is dominant afterward, we should
not decay $\gamma_{1}$ at the first BN. For convenience, we also call this
parameter $\gamma_{down}$.

Figure 4: Early stage in (left) PreActResNet and (right) original ResNet (Case
4).

#### Case 4. Early stage in ResNet

The input variance of the first residual block is determined by the early
stage of ResNet before the residual block starts. The early stage of ResNet,
which is also called the stem, consists of convolution, BN, ReLU, and pooling
layers without a skip connection in the original ResNet (Figure 4, Right).
Ignoring pooling, from the output of early stage
$x_{1}=ReLU(B_{0}(W_{0}x_{0}))$, we have

$\displaystyle Var[x_{1,i}]$
$\displaystyle=\frac{1}{2}E[B_{0,i}(W_{0}x_{0}))^{2}]$ (18)
$\displaystyle=\frac{1}{2}\gamma_{0,i}^{2}.$ (19)

Therefore, $\gamma_{0}$ _sets_ the output variance of the early stage. The
output of the early stage becomes the input of the residual block of the first
stage. Because a large variance at the stage starting point is favorable,
$\gamma_{0}$ should be large. In this regard, we present the following
guideline:

#### Guideline 3.

Because the early stage plays the role of variance set, we should not decay
$\gamma_{0}$.

In case of PreActResNet, BN is not applied in the early stage (Figure 4,
Left). Thus, we do not have to consider $\gamma_{0}$.

Finally, in the residual branch, additional $\gamma$ parameters exist in the
BN before the last BN for both PreActResNet and the original ResNet. We call
those parameters $\gamma_{others}$. Because they do not affect the variance of
the residual branches, our variance analysis cannot explain their role. The
$\gamma_{others}$ parameters are discussed again in Section 3; here, we simply
present the conclusion:

#### Guideline 4.

Though other BNs do not determine the variance, $L_{2}$ regularization on
$\gamma_{others}$ helps optimization by improving the effective learning rate.
Thus, we should decay $\gamma_{others}$.

Figure 5: Variance of each feature map of ResNet-50 pre-trained on ImageNet.

Further, using ResNet-50, we measured the variance of each feature map (Figure
5). First, the variance is (re)set at layer 1-0, and then the variance
decreases at layer 1-1, which is expected from the half variance accumulation
(Eqs. 10 and 11). In layer 2-0, where the subsequent stage begins, the
variance is reset to a new larger value and decreases again from layer 2-1 to
layer 2-3. As such, both the variance reset to a larger value and half
variance accumulation with decreasing value appear in each residual block,
which agrees with our claim.

#### Case 5. Transformer block

The transformer is attracting attention for its high performance in various
tasks, including computer vision [5] and natural language processing [20, 4].
The transformer uses a series of transformer blocks, which consists of
residual branches with self-attention and multilayer perceptron blocks. In
contrast to ResNet, the transformer uses LayerNorm (LN) [1]. However, LN also
rescales normalized features using $\gamma$ and $\beta$. Therefore, our
variance analysis can explain how the $\gamma$ parameter of LN affects the
variance within the transformer block. See the Appendix for the mathematical
details regarding the roles of $\gamma_{1}$ and $\gamma_{2}$ in the first and
second LNs. Here, we simply present the following guideline for the
transformer block:

#### Guideline 1T.

Because a transformer block accumulates the variance, we should decay
$\gamma_{1}$ and $\gamma_{2}$.

Note that the transformer has no downsampling block (and hence there is no
Guideline 2T). Further, some transformers such as bidirectional encoder
representations from transformers (BERT) [4] have LN in the early stage, whose
$\gamma_{0}$ should not be subjected to the $L_{2}$ regularization (Guideline
3T). Finally, some transformers [20] have another LN between the final
transformer block and the head layer. The role of the $\gamma_{other}$
parameters of LN cannot be explained by our variance analysis, but in the same
context as Guideline 4, $L_{2}$ regularization on $\gamma_{other}$ helps
optimization (Guideline 4T).

##  3 Effective learning rate analysis

In this section, we explore why $\gamma_{others}$ parameters should be decayed
even though their scale does not affect the BN output. Reference [19, 11, 26]
argued that in a neural network with BN, the weight scale does not affect the
BN output. They introduced the _effective learning rate_ to claim that a
smaller weight scale is advantageous for optimization, which can be achieved
through $L_{2}$ regularization on weights. In our paper, we note that $\gamma$
is a trainable parameter in a neural network similar to weights. Extending
their claim, we investigate the effective learning rate for the optimization
of $\gamma$.

BN is composed of a normalization step $N(\cdot)$ and linear scaling step
$L(\cdot)$. From a series of [Weight–BN–ReLU] blocks, we decompose BN into
[N–L] and investigate the intermediate [L–ReLU–Weight–N] block. In the $l$-th
intermediate block, weight layer holds $W_{l}$ and $L$ step holds
$\Gamma_{l}$, where $\Gamma_{l}$ is a diagonal matrix with
$(\Gamma_{l})_{i,i}=\gamma_{i}$ and $(\Gamma_{l})_{i,j}=0$ for $i\neq j$. From
the input feature map $x_{l}$, the intermediate block outputs
$x_{l+1}=y(x_{l};W_{l},\Gamma_{l})=N[W_{l}ReLU(\Gamma_{l}x_{l})]$.

Here, we decompose the diagonal matrix into its norm and direction, i.e.,
$\Gamma_{l}=\left\lVert\Gamma_{l}\right\rVert\hat{\Gamma}_{l}$. Then,

$\displaystyle y(x_{l};W_{l},\Gamma_{l})$
$\displaystyle=N[W_{l}ReLU(\left\lVert\Gamma_{l}\right\rVert\hat{\Gamma}_{l}x_{l})]=N[W_{l}\left\lVert\Gamma_{l}\right\rVert
ReLU(\hat{\Gamma}_{l}x_{l})]$ (20)
$\displaystyle=N[W_{l}ReLU(\hat{\Gamma}_{l}x_{l})].$ (21)

Thus, the scale of $\gamma$ in BN does not affect the intermediate block
output. If we investigate the gradient of $y(x_{l};W_{l},\Gamma_{l})$ with
respect to $\Gamma_{l}$, we obtain

$\displaystyle\nabla_{\Gamma_{l}}y(x_{l};W_{l},\Gamma_{l})=\frac{1}{\left\lVert\Gamma_{l}\right\rVert}\nabla_{\hat{\Gamma}_{l}}y(x_{l};W_{l},\hat{\Gamma}_{l}).$
(22)

This scale-variant gradient property was used in [19, 11] to claim that
$L_{2}$ regularization on weights helps optimization even in neural networks
with BN. Thus, effective learning rate analysis is expected to be applicable
to $\gamma$ in BN. From Eq. 22, we have
$\nabla_{\Gamma_{l}}L(\Gamma_{l})=\frac{1}{\left\lVert\Gamma_{l}\right\rVert}\nabla_{\hat{\Gamma}_{l}}L(\hat{\Gamma}_{l})$
for loss function $L$. First, gradient descent with respect to $\Gamma_{l}$ is
in the form of

$\displaystyle\Gamma_{l,t+1}=\Gamma_{l,t}-\eta\nabla_{\Gamma_{l,t}}L(\Gamma_{l,t}).$
(23)

If we investigate gradient descent with respect to $\hat{\Gamma}_{l}$, we
have,

$\displaystyle\hat{\Gamma}_{l,t+1}=\hat{\Gamma}_{l,t}-\left(\eta\left\lVert\Gamma_{l,t}\right\rVert^{-2}\right)\nabla_{\hat{\Gamma}_{l,t}}L(\hat{\Gamma}_{l,t}).$
(24)

Figure 6: Norm of the first update by the initial value of $\gamma$.

Here, $\eta\left\lVert\Gamma_{l,t}\right\rVert^{-2}$ is called the effective
learning rate [19, 11]. If the scale of $\gamma$ in BN decreases, the
effective learning rate increases, which prevents the optimization from
saturating due to the large weight norm. Although we followed the derivation
in [19], the derivation in [11] shows similar results. Thus, although $L_{2}$
regularization on $\gamma_{others}$ does not affect the variance, it improves
the effective learning rate and optimization. In this regard, we advocate
applying $L_{2}$ regularization to $\gamma_{others}$.

Now, we empirically validate Eq. 24. Using ResNet-18, we varied the initial
value of $\gamma$ and measured the norm of the first update,
$\lVert\hat{\Gamma}_{l,1}-\hat{\Gamma}_{l,0}\rVert$ (Figure 6). From their log
values, a regression fit yielded a coefficient near $-2$, which agrees with
$\lVert\hat{\Gamma}_{l,t+1}-\hat{\Gamma}_{l,t}\rVert\propto\lVert\Gamma_{l,t}\rVert^{-2}$.

Note that this effective learning rate analysis is applicable to the $\gamma$
parameters of all four categories. Although applying $L_{2}$ regularization on
$\gamma_{down}$ and $\gamma_{0}$ is harmful to variance control, it could be
advantageous for improving optimization with respect to the effective learning
rate.

##  4 Experiments

###  4.1 Image classification tasks

We tested whether Guidelines 1–4 are valid in practical tasks. First, in the
training task of ResNet and its variants [8, 9, 25, 23], we measured the
performance of applying $L_{2}$ regularization to the weights and $\gamma$ of
each group with $\frac{1}{2}\lambda(\left\lVert
W\right\rVert^{2}_{2}+\left\lVert\gamma\right\rVert^{2}_{2})$. For all the
experiments, we report the average results from three experiments. For
hyperparameters and training details, refer to the Appendix.

Table 2: Test accuracy (%) for the PET dataset. $\boldsymbol{L_{2}}$ Regularization | $\boldsymbol{W}$ | $\boldsymbol{W,\gamma_{last}}$ | $\boldsymbol{W,\gamma_{down}}$ | $\boldsymbol{W,\gamma_{0}}$ | $\boldsymbol{W,\gamma_{others}}$
---|---|---|---|---|---
ResNet-18 | 83.5733 | 85.4100 | 82.5800 | 82.8800 | 84.6000
|  | (+1.8367) | (-0.9933) | (-0.6933) | (+1.0267)
ResNet-50 | 82.9433 | 85.6500 | 80.0833 | 82.4000 | 83.5133
|  | (+2.7067) | (-2.8600) | (-0.5433) | (+0.5700)
ResNet-101 | 82.0100 | 84.4200 | 78.9400 | 81.3767 | 83.1200
|  | (+2.4100) | (-3.0700) | (-0.6333) | (+1.1100)
ResNet-152 | 80.9867 | 85.7400 | 79.2100 | 82.3700 | 82.9100
|  | (+4.7533) | (-1.7767) | (+1.3833) | (+1.9233)
Wide ResNet-50-2 | 82.9433 | 85.6500 | 80.0833 | 82.4000 | 83.5133
|  | (+2.7067) | (-2.8600) | (-0.5433) | (+0.5700)
Wide ResNet-101-2 | 82.4600 | 85.1400 | 81.2000 | 83.4533 | 84.6300
|  | (+2.6800) | (-1.2600) | (+0.9933) | (+2.1700)
ResNeXt-50-32x4d | 84.4167 | 85.6200 | 84.1133 | 84.6000 | 85.7400
|  | (+1.2033) | (-0.3033) | (+0.1833) | (+1.3233)
ResNeXt-101-32x8d | 84.0233 | 85.4100 | 83.6067 | 82.9700 | 85.2900
|  | (+1.3867) | (-0.4167) | (-1.0533) | (+1.2667)
Our Guidelines | ✓ | ✓ | ✗ | ✗ | ✓

Table 2 shows image classification accuracy on the Oxford-IIIT PET dataset
[16] for different $L_{2}$ regularization setups. First, when the
$\gamma_{last}$ parameters were subjected to $L_{2}$ regularization, the test
accuracy increased approximately 1%–4% compared to the case where $L_{2}$
regularization was not applied to them. This is consistent with Guideline 1,
which states that we should decay the $\gamma_{last}$. However, when $L_{2}$
regularization was applied to $\gamma_{down}$, the test accuracy decreased
approximately 1%–3%. This is consistent with Guideline 2, which states that
$L_{2}$ regularization for $\gamma_{down}$ is undesirable. Similarly, for
$\gamma_{0}$ and $\gamma_{others}$, the experimental results agree with
Guidelines 3 and 4.

#### Limitations

However, for $\gamma_{0}$, an increase in performance was observed in some
models. When the effect of improving the effective learning rate is more
dominant than the variance control, it can result in performance improvement.
Because the first stage is relatively shallow compared to other stages, the
variance from $\gamma_{0}$ would have a minor effect, and improvement of the
effective learning rate may become dominant. The coexistence of variance
control and improved effective learning rate requires further research. See
the Appendix for more results on other datasets, including NABirds, Food-101,
and CIFAR-10.

###  4.2 Natural language processing tasks

We also tested whether our proposed guidelines are valid for transformers on
natural language processing tasks. First, we visited a machine translation
task using a transformer. The BLEU score [15] was measured for the German to
English translation task using the IWSLT-14 dataset [2] (Table 3). We measured
the performance before and after applying $L_{2}$ regularization to
$\gamma_{1}$ and $\gamma_{2}$. For both cases, an increase in the BLEU score
was observed. This result is consistent with Guideline 1T, which states that
small $\gamma$ is desirable for transformer blocks because they accumulate
variance.

Table 3: BLEU scores (%) for the IWSLT-14 dataset. $\boldsymbol{L_{2}}$ Regularization | $\boldsymbol{W}$ | $\boldsymbol{W,\gamma_{1}}$ | $\boldsymbol{W,\gamma_{2}}$
---|---|---|---
Transformer | 34.7894 | 34.9063 | 35.1385
|  | (+0.1170) | (+0.3491)

Our claim was also verified in the text classification task using BERT for the
SST-2 task from GLUE [21]. Following Guideline 1T improved test accuracy here
as well (Table 4).

Table 4: Test accuracy (%) for the GLUE SST-2 task. $\boldsymbol{L_{2}}$ Regularization | $\boldsymbol{W}$ | $\boldsymbol{W,\gamma_{1},\gamma_{2}}$ | Difference
---|---|---|---
BERT | 91.8807 | 92.0872 | +0.2064

##  5 Conclusion

In this paper, we studied $L_{2}$ regularization for the $\gamma$ parameters
of BN. We theoretically showed that this $\gamma$ should be controlled so that
the residual blocks behave similarly to identity mapping. Specifically, we
discussed the cases in which it is desirable to decay the $\gamma$ parameters
of BN and the cases in which it is not, presenting four guidelines for their
management. The proposed guidelines were verified in experiments on several
variants of residual networks and transformer models. We confirmed that the
performance increases when the guidelines are followed and decreases when the
guidelines are violated.

## References

  * [1] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer Normalization.  CoRR, abs/1607.06450, 2016.
  * [2] Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico.  Report on the 11th IWSLT evaluation campaign.  In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, 2014.
  * [3] Soham De and Samuel L. Smith.  Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks.  In NeurIPS, 2020.
  * [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In NAACL-HLT, 2019.
  * [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In ICLR, 2021.
  * [6] Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.  CoRR, abs/1706.02677, 2017.
  * [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.  In ICCV, 2015.
  * [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In CVPR, 2016.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity Mappings in Deep Residual Networks.  In ECCV, 2016.
  * [10] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li.  Bag of Tricks for Image Classification with Convolutional Neural Networks.  In CVPR, 2019.
  * [11] Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry.  Norm matters: efficient and accurate normalization schemes in deep networks.  In NeurIPS, 2018.
  * [12] Sergey Ioffe and Christian Szegedy.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.  In ICML, 2015.
  * [13] Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, Tiegang Chen, Guangxiao Hu, Shaohuai Shi, and Xiaowen Chu.  Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes.  CoRR, abs/1807.11205, 2018.
  * [14] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  ImageNet Classification with Deep Convolutional Neural Networks.  In NIPS, 2012.
  * [15] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a Method for Automatic Evaluation of Machine Translation.  In ACL, 2002.
  * [16] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In CVPR, 2012.
  * [17] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller.  Striving for Simplicity: The All Convolutional Net.  In ICLR, 2015.
  * [18] Cecilia Summers and Michael J. Dinneen.  Four things everyone should know to improve batch normalization.  In ICLR, 2020.
  * [19] Twan van Laarhoven.  L2 Regularization versus Batch and Weight Normalization.  CoRR, abs/1706.05350, 2017.
  * [20] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In NIPS, 2017.
  * [21] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.  In ICLR, 2019.
  * [22] Yuxin Wu and Kaiming He.  Group Normalization.  Int. J. Comput. Vis., 2020.
  * [23] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated Residual Transformations for Deep Neural Networks.  In CVPR, 2017.
  * [24] Junjie Yan, Ruosi Wan, Xiangyu Zhang, Wei Zhang, Yichen Wei, and Jian Sun.  Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization.  In ICLR, 2020.
  * [25] Sergey Zagoruyko and Nikos Komodakis.  Wide Residual Networks.  In BMVC, 2016.
  * [26] Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger B. Grosse.  Three Mechanisms of Weight Decay Regularization.  In ICLR, 2019.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes] We discussed limitations
and future research in the third paragraph of Section 4

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [Yes]
See Section 2.

    2. (b)

Did you include complete proofs of all theoretical results? [Yes] See Section
2 and the Appendix.

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
See the supplemental material.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] See the Appendix.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [No] Though we did not use error bars, we report
the average from three results for all experiments.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the
Appendix.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] The data
and models are properly cited.

    2. (b)

Did you mention the license of the assets? [No] Though we did not mention the
licenses, we used existing assets, which are available online and do not have
any license issues.

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes] See the supplemental material.

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [No] We used existing assets which are available online
and do not have any issues, such as the PyTorch library and open datasets.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [No] Though we did not discuss
such issues, the assets we used do not have any critical issues such as
information about a specific person.

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]