# "An Overview of Initialization for Deep Neural Networks, Part 1"

> "Without appropriate choice of activation function and/or smart initialization for it neural networks, especially deep ones, tend to refuse training at all. We will consider sigmoid and tanh activation functions and two initialization methods most suitable for them - LeCun and Xavier (Glorot) initialization."

- toc: false
- branch: master
- badges: false
- comments: false
- categories: [fastpages, jupyter]
- use_math: true

For the past few years deep neural networks have proven themselves to be able to solve various AI-level tasks effectively. In some very specific tasks, such as image classification, game of Go, autonomous driving, and many others they already perform better than humans. These achievements were not possible without tremendous research invested in study of difficulties that occur during training of deep networks, and methods to overcome them. A part of this progress is due to better non-linearities and initialization methods.

## What Is the Choice of Activation Function and Initialization and Why Is It Important?


Neural networks are typically trained via stochastic gradient descent or one of its more powerful variants, such as RMSprop or Adam, that use gradients computed by the back propagation algorithm. The back propagation algorithm essentially recursively applies the chain rule of calculus to compute gradients efficiently. Just as a reminder, the chain rule of calculus look like this

\begin{equation}
    \frac{\partial}{\partial x}g(f(x)) = \frac{\partial g}{\partial f} \frac{\partial f}{\partial x}
\end{equation}

In earlier years folks used sigmoid activation function, defined as

\begin{equation}
    \sigma(x) = \frac{1}{1 + e^{-x}}
\end{equation}

However, soon it became evident that such nets are hard to train, because of some properties of sigmoid. One such property is the effect of saturation. Look at the graph of sigmoid.


```
def sigmoid(x):
    return 1. / (1. + np.exp(-x))

def sigmoid_derivative(x):
    return sigmoid(x) * (1. - sigmoid(x))

def plot_activation_and_derivative(activation, derivative, inputs, titles):
    fig, (ax_f, ax_d) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
    ax_f.plot(inputs, activation(inputs))
    ax_f.set_title(titles[0])
    ax_d.plot(inputs, derivative(inputs))
    ax_d.set_title(titles[1])
    plt.show()

values = np.arange(-6.0, 6.0, 0.01)
plot_activation_and_derivative(sigmoid, sigmoid_derivative, values,
                               ('Sigmoid activation', 'Derivative of sigmoid'))
```

When input values become large enough (whether positive or negative), output of sigmoid remains roughly unchanged and goes to 1 in case of positive inputs, or to 0 in case of negative. This saturation of sigmoid causes gradient vanishing. Since output values are roughly equal in those saturated tails, the derivative of sigmoid becomes very close to zero. As back propagation progresses towards input layers, gradients get multiplied, and if at least one of multiples becomes close to zero, the whole product is likely to become close to zero as well. Hence it can require too many gradient updates to tweak parameters in right way. Now look at the graph of the derivative of sigmoid more closely. You can note that its maximum value is 0.25 when input value is 0. So, in the best case, gradient gets four times smaller after sigmoid activation during back propagation.

There is another saturating activation function that, however, suffers less from vanishing gradient problems. It is called hyperbolic tangent, or tanh for short.

\begin{equation}
    \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}
\end{equation}

Actually it can be shown from their definitions that tanh can be expressed in terms of sigmoid as follows

\begin{equation}
    \tanh(x) = 2\sigma(2x) - 1
\end{equation}

In spite of that, tanh has two advantages over the sigmoid. First, it is symmetric in the origin, which makes it easier to learn identity function, and second, maximum value of its derivative is 1.


```
def tanh(x):
    return np.tanh(x)

def tanh_derivative(x):
    return 1. / np.cosh(x) ** 2

plot_activation_and_derivative(tanh, tanh_derivative, values,
                               ('Tanh activation', 'Tanh derivative'))
```

Tanh is widely used in recurrent neural networks today.

Nowadays we have many ways to address vanishing gradient and similar exploding gradient issues. And one of these is better initialization. Initialization is also important because gradient-based methods can stuck in bad local minima, not reaching the global minimum, and take long to converge because of plateus in the cost function surface. Better initialization reduces chances to start from "unlucky" points.

## LeCun Initialization

This is one of the first heuristics for initialization. It was proposed in 1998 by Yann LeCun et al. in the paper called "[Efficient BackProp](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf)". LeCun initialization suggests that weights of network are drawn from normal distribution of the following form

\begin{equation}
    W \sim \mathcal{N}\Bigg(0, \frac{1}{fan_{in}}\Bigg)
\end{equation}

where $fan_{in}$ is the number of connections feeding into the node. This is also known as LeCun normal initialization, while in case of LeCun uniform initialization weights are drawn from uniform distribution

\begin{equation}
    W \sim \mathcal{U}\Bigg(-\frac{1}{\sqrt{fan_{in}}}, \frac{1}{\sqrt{fan_{in}}}\Bigg)
\end{equation}

LeCun initialization was designed specifically for sigmoid activation. But it has more applications than that. It was proposed to be used with the [SELU activation function and self-normalizing networks](https://arxiv.org/pdf/1706.02515.pdf).

Note that regardless the method of initialization, weights are usually drawn either from normal or uniform distribution, and there's connection between them. If we initialize from a normal distribution with $\mathrm{Var}[W] = \sigma^2$, we can easily switch to uniform distribution by computing $a = \sqrt{3\sigma^2}$ and sampling weights from $\mathcal{U}(-a, a)$. Because of this connection I will only consider normal distribution and specify suggested variance for it.

### Derivation of LeCun Initialization

The reasoning behind LeCun initialization is fairly simple. We don't want weights to be very large, because they will lead to saturation and cause gradient vanishing. We also don't want weights to be very small, because they will make gradient small as well and make the whole optimization slow. What we want is that initial weights map inputs into linear region of sigmoid (roughly between -2 and 2), so the gradient is large enough and network learns easier linear part first.

Consider a sigmoid layer and let $x$ denote input example and $y$ linearly transformed input, so

\begin{equation}
    y = xW \\
    y_i = \sum_{j = 1}^{fan_{in}}{x_{j}w_{ji}}
\end{equation}

We can think of bias term as weight that always receives 1 as input and omit it. To achieve the above requirements we first demand $x$ to have zero mean and variance of 1. This is satisfied by normalizing inputs. Now we want initialize weights so that they ensure $y$ also has zero mean and unit variance. Let's write down expression for the variance of $y_i$

\begin{equation}
    \mathrm{Var}[y_i] = \mathrm{Var}\Bigg[\sum_{j = 1}^{fan_{in}}{x_{j}w_{ji}}\Bigg] = \sum_{j = 1}^{fan_{in}} w_{ji}^2\mathrm{Var}[x_j] + \sum_{j \neq k}w_{ji}w_{ki}\mathrm{Cov}[x_j, x_k]
\end{equation}

Assuming uncorrelated $x$, we have $\mathrm{Cov}[x_j, x_k] = 0$ for all pairs $j \neq k$. Recall we assumed all $\mathrm{Var}[x_i] = 1$. Hence

\begin{equation}
    \mathrm{Var}[y_i] = \sum_{j = 1}^{fan_{in}} w_{ji}^2 \Rightarrow\ \mathrm{Var}[w] = \frac{\mathrm{Var}[y]}{fan_{in}} = \frac{1}{fan_{in}}
\end{equation}

And that's it! One thing that should be mentioned here is that original paper also proposed a variant of sigmoid that ensures $\sigma(y)$ preserves zero mean and unit variance of $y$.

## Xavier (Glorot) Initialization

The next method was proposed in paper "[Understanding the Difficulty of Training Deep Feedforward Neural Networks](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)" by Xavier Glorot and Yoshua Bengio in 2010. I don't know who decides whether we should call an invention after author's first name or last name, but this technique is widely known under both Xavier initialization and Glorot initialization.

If you have skimmed through the derivation of LeCun initialization above or read it in original paper, you might have noticed that it was motivated entirely from the perspective of the forward pass. In their paper Glorot and Bengio additionly require preserving variance of gradient during the backward pass. From one hand, the forward pass requires, as previously

\begin{equation}
    \mathrm{Var}[W] = \frac{1}{fan_{in}}
\end{equation}

They show that the backward pass from the other hand requires

\begin{equation}
    \mathrm{Var}[W] = \frac{1}{fan_{out}}
\end{equation}

where $fan_{out}$ is the number of connections going out of a unit.

Both constraints can be satisfied if and only if $fan_{in} = fan_{out}$, which isn't always the case. As a compromise authors suggest to set variance to

\begin{equation}
    \mathrm{Var}[W] = \frac{1}{fan_{avg}} = \frac{2}{fan_{in} + fan_{out}}
\end{equation}

Observe also that when $fan_{in} = fan_{out}$ Xavier initialization and LeCun initialization are equivalent.

The paper compares performance of several networks of the same architecture that only differ in activation function and initialization method. Those networks were trained on different data sets using the same procedure. The final test errors for networks using tanh activation are represented in the following table

TYPE | Shapeset | MNIST | CIFAR-10 | Small ImageNet
--- | --- | --- | --- | ---
Tanh | 27.15 | 1.76 | 55.9 | 70.58
Tanh N | 15.60 | 1.64 | 52.92 | 68.57

Here tanh stands for the network with tanh activations and LeCun uniform initialization, while tanh N - for the network with tanh activations and Xavier uniform initialization. As can be seen, on all data sets Xavier initialization gives little to significant boost in performance when compared to LeCun initialization.

A trouble with Xavier initialization is that it assumes linear units. This is a valid assumption when inputs are mapped to linear parts of non-linearities, but invalid in general. Nevertheless it still works pretty well. The following histograms of activation values and gradients from the paper show that Xavier initialization in experimental settings works as intended. Check out details in original paper!

### Derivation of Xavier Initialization

Consider a neural network with $L$ consequent layers. We begin with writing down equations for both forward and backward propagations. We make the same assumptions about input data as in the derivation of LeCun initialization. The forward pass is given by

\begin{equation}
    Z^{[i]}_k = \varphi^{[i]}(Z^{[i - 1]}_k) W^{[i]}_{:, k}
\end{equation}

And the backward pass by

\begin{equation}
    \frac{\partial\mathscr{L}}{\partial Z^{[i]}_k} = \varphi^{[i]\prime}(Z^{[i]}_k)W^{[i]}_{k, :}\frac{\partial\mathscr{L}}{\partial Z^{[i + 1]}} \\
    \frac{\partial\mathscr{L}}{\partial W^{[i]}_{lk}} = Z^{[i]}_l \frac{\partial\mathscr{L}}{\partial Z^{[i]}_k}
\end{equation}

Then for a layer number $i$, assuming that $\varphi^{[i]\prime}(z^{[i]}_k) = 1$

\begin{equation}
    \mathrm{Var}[Z^{[i]}] = \mathrm{Var}[X]\prod_{j=0}^{i-1}n^{[j]}\mathrm{Var}[W^{[j]}]\\
    \mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[i]}}\Bigg] = \mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[L]}}\Bigg]\prod_{j = i}^{L - 1}n^{[j + 1]}\mathrm{Var}[W^{[j]}]\\
    \mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial W^{[i]}}\Bigg] = \mathrm{Var}[Z^{[i]}]\mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[i]}}\Bigg]
\end{equation}

From the perspective of the forward pass we wish

\begin{equation}
    \forall\ (i, j):\ \mathrm{Var}[Z^{[i]}] = \mathrm{Var}[Z^{[j]}]
\end{equation}

And from the back prop point of view we want

\begin{equation}
    \forall\ (i, j):\ \mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[i]}}\Bigg] = \mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[j]}}\Bigg]
\end{equation}

These two constraints can be easily satisfied if

\begin{equation}
    \forall\ i:\ n^{[i]}\mathrm{Var}[W^{[i]}] = \mathrm{Var}[X] = 1\\
    \forall\ i:\ n^{[i + 1]}\mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[i]}}\Bigg] = \mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[L]}}\Bigg] = 1
\end{equation}

From these two equations we finally get

\begin{equation}
    \mathrm{Var}[W^{[i]}] = \frac{1}{n^{[i]}} = \frac{1}{fan_{in}} \\
    \mathrm{Var}[W^{[i]}] = \frac{1}{n^{[i + 1]}} = \frac{1}{fan_{out}}
\end{equation}

As a compromise for cases when $fan_{in} \neq fan_{out}$

\begin{equation}
    \mathrm{Var}[W^{[i]}] = \frac{2}{fan_{in} + fan_{out}}
\end{equation}

## Conclusions

Without appropriate choice of activation function and/or smart initialization for it neural networks, especially deep ones, tend to refuse training at all. Better activation functions and better initialization strategies are tightly related, so that they complement each other and evolve together. Once popular sigmoid non-linearity turned out to be impractical to use in hidden layers, and now is only used in the output layer for binary classification, while tanh that can be expressed in terms of sigmoid, is still useful for hidden layers. At the first glance initialization strategies seem some wisdom sent from heaven, but at least in case of LeCun and Xavier initialization their derivations are in fact straightforward. All you need to know is where to start and what assumptions to make.

050ca72f7fa9139dfde41ee062e4bce84f9f437c
ipynb
Jupyter Notebook
_notebooks/2020-07-31-An-Overview-of-Activations-and-Initializations-for-Deep-Neural-Networks,-Part-1.ipynb
akulchik/blog
3acb74d4943fb21064f4033bc7ac7a4044ce306d
Apache-2.0
2021-02-12T19:33:47.000Z
2021-02-12T19:33:47.000Z
_notebooks/2020-07-31-An-Overview-of-Activations-and-Initializations-for-Deep-Neural-Networks,-Part-1.ipynb
akulchik/blog
3acb74d4943fb21064f4033bc7ac7a4044ce306d
Apache-2.0
2021-03-30T12:28:14.000Z
2021-09-28T05:26:49.000Z
_notebooks/2020-07-31-An-Overview-of-Activations-and-Initializations-for-Deep-Neural-Networks,-Part-1.ipynb
akulchik/blog
3acb74d4943fb21064f4033bc7ac7a4044ce306d
Apache-2.0
Qwen/Qwen-72B
1. YES
2. YES