# Weight Intilization
Weight and bias intialization is one of the important factor responsible for today's state of the art algorithm.

Weights intialization is done in random fashion but that randomness is to be tuned in various ways to get optimum result.

<p><strong>Pitfall: all zero initialization</strong>. Lets start with what we should not do. Note that we do not know what the final value of every weight should be in the trained network, but with proper data normalization it is reasonable to assume that approximately half of the weights will be positive and half of them will be negative. A reasonable-sounding idea then might be to set all the initial weights to zero, which we expect to be the “best guess” in expectation. This turns out to be a mistake, because if every neuron in the network computes the same output, then they will also all compute the same gradients during backpropagation and undergo the exact same parameter updates. In other words, there is no source of asymmetry between neurons if their weights are initialized to be the same.</p>

<p><strong>Vanishing Gradient</strong> In deep nueral network if <b>sigmoid</b> neuron is used there will be a problem of vanishing gradient because the value of sigmoid is between 0 and 1 and derivative of sigmoid is $$sigmoid(x)*(1-sigmoid(x))$$ and  when gradient flows backward in the network it is multiplied by subsequent wights which decreases the gradient value so the gradient to the beginning layers is very close to zero that means there is no learning. Using ReLU solves this problem.
    
<p><strong>Exploding Gradient</strong>:- Vanishing gradient is not much of a problem but in the contrary Exploding gradient is something on which we can ponder. This problem occours when weight are intialized with huge number like 10 or 100 and input to the network is not normalized. In this scenario either of these three can happen.
    
    1. The model is unable to get traction on your training data (e.g. poor loss).
    2. The model is unstable, resulting in large changes in loss from update to update.
    3. The model loss goes to NaN during training.
    
To Solve this problem the solution given are:
 1. Redesign the network
In deep neural networks, exploding gradients may be addressed by redesigning the network to have fewer layers.
There may also be some benefit in using a smaller batch size while training the network.

2. Weight Regularizers
Another approach, if exploding gradients are still occurring, is to check the size of network weights and apply a penalty to the networks loss function for large weight values.

This is called weight regularization and often an L1 (absolute weights) or an L2 (squared weights) penalty can be used.

3. Gradient Clipping
Clip the gradient before updating the weights. Either the gradient can be clipped by using l2 norm or the easiest way to clip is by values. we define a threshold value with range `[-limit ,limit]` if the grad value exceeds the limit we replace it with the limit value 

<p><strong>Small random numbers</strong>. Therefore, we still want the weights to be very close to zero, but as we have argued above, not identically zero. As a solution, it is common to initialize the weights of the neurons to small numbers and refer to doing so as <em>symmetry breaking</em>. The idea is that the neurons are all random and unique in the beginning, so they will compute distinct updates and integrate themselves as diverse parts of the full network. The implementation for one weight matrix might look like <code class="highlighter-rouge">W = 0.01* np.random.randn(D,H)</code>, where <code class="highlighter-rouge">randn</code> samples from a zero mean, unit standard deviation gaussian. With this formulation, every neuron’s weight vector is initialized as a random vector sampled from a multi-dimensional gaussian, so the neurons point in random direction in the input space. It is also possible to use small numbers drawn from a uniform distribution, but this seems to have relatively little impact on the final performance in practice.</p>

<p><em>Warning</em>: It’s not necessarily the case that smaller numbers will work strictly better. For example, a Neural Network layer that has very small weights will during backpropagation compute very small gradients on its data (since this gradient is proportional to the value of the weights). This could greatly diminish the “gradient signal” flowing backward through a network, and could become a concern for deep networks.</p>

<p><strong>Calibrating the variances with 1/sqrt(n) Xavier/Glorot Initialization</strong>. One problem with the above suggestion is that the distribution of the outputs from a randomly initialized neuron has a variance that grows with the number of inputs. It turns out that we can normalize the variance of each neuron’s output to 1 by scaling its weight vector by the square root of its <em>fan-in</em> (i.e. its number of inputs). That is, the recommended heuristic is to initialize each neuron’s weight vector as: <code class="highlighter-rouge">w = np.random.randn(n) / sqrt(n)</code>, where <code class="highlighter-rouge">n</code> is the number of its inputs. This ensures that all neurons in the network initially have approximately the same output distribution and empirically improves the rate of convergence.</p>

<p>The sketch of the derivation is as follows: Consider the inner product \(s = \sum_i^n w_i x_i\) between the weights \(w\) and input \(x\), which gives the raw activation of a neuron before the non-linearity. We can examine the variance of \(s\):</p>


\begin{align}
\text{Var}(s) &= \text{Var}(\sum_i^n w_ix_i) \\\\
&= \sum_i^n \text{Var}(w_ix_i) \\\\
&= \sum_i^n [E(w_i)]^2\text{Var}(x_i) + E[(x_i)]^2\text{Var}(w_i) + \text{Var}(x_i)\text{Var}(w_i) \\\\
&= \sum_i^n \text{Var}(x_i)\text{Var}(w_i) \\\\
&= \left( n \text{Var}(w) \right) \text{Var}(x)
\end{align} 

<p>where in the first 2 steps we have used <a href="http://en.wikipedia.org/wiki/Variance">properties of variance</a>. In third step we assumed zero mean inputs and weights, so \(E[x_i] = E[w_i] = 0\). Note that this is not generally the case: For example ReLU units will have a positive mean. In the last step we assumed that all \(w_i, x_i\) are identically distributed. From this derivation we can see that if we want \(s\) to have the same variance as all of its inputs \(x\), then during initialization we should make sure that the variance of every weight \(w\) is \(1/n\). And since \(\text{Var}(aX) = a^2\text{Var}(X)\) for a random variable \(X\) and a scalar \(a\), this implies that we should draw from unit gaussian and then scale it by \(a = \sqrt{1/(n)}\), to make its variance \(1/n\). This gives the initialization <code class="highlighter-rouge">w = np.random.randn(n) / sqrt(n<sub>in</sub>+n<sub>out</sub>)</code>.</p>


<p><strong>Sparse initialization</strong>. Another way to address the uncalibrated variances problem is to set all weight matrices to zero, but to break symmetry every neuron is randomly connected (with weights sampled from a small gaussian as above) to a fixed number of neurons below it. A typical number of neurons to connect to may be as small as 10.</p>

<p><strong>Initializing the biases</strong>. It is possible and common to initialize the biases to be zero, since the asymmetry breaking is provided by the small random numbers in the weights. For ReLU non-linearities, some people like to use small constant value such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning and therefore obtain and propagate some gradient. However, it is not clear if this provides a consistent improvement (in fact some results seem to indicate that this performs worse) and it is more common to simply use 0 bias initialization.</p>

<p><strong>In practice</strong>, the current recommendation is to use ReLU units and use the <code class="highlighter-rouge">w = np.random.randn(n) * sqrt(2.0/n)</code>, as discussed in <a href="http://arxiv-web3.library.cornell.edu/abs/1502.01852">He et al.</a>.</p>

Pytorch Supports various popular intialization type which can seen here --->> https://pytorch.org/docs/stable/nn.html#torch-nn-init




```python
from torch import nn

# a simple network
model=nn.Sequential(nn.Linear(2, 5),
                         nn.ReLU(),)
next(model.parameters())
```




    Parameter containing:
    tensor([[-0.4624,  0.1237],
            [ 0.3361,  0.3213],
            [ 0.0601, -0.5251],
            [ 0.0932, -0.6454],
            [-0.1530,  0.3163]], requires_grad=True)




```python

# initialization function, first checks the module type,
# then applies the desired changes to the weights
def init_normal(m):
    if type(m) == nn.Linear:
        nn.init.uniform_(m.weight)
    elif isinstance(m , nn.Conv2d):
        nn.init.kaiming_normal_(m.weight)

# use the modules apply function to recursively apply the initialization to every submodule.
model.apply(init_normal)
next(model.parameters())
```




    Parameter containing:
    tensor([[0.9828, 0.0986],
            [0.8059, 0.4303],
            [0.2424, 0.9537],
            [0.5189, 0.4018],
            [0.1175, 0.8017]], requires_grad=True)



# Additional Resources

https://pytorch.org/docs/stable/nn.html#torch-nn-init

http://cs231n.github.io/neural-networks-2/#init


```python

```

86bd3f1db3e049c786227cc2cf68e58c2c3257dd
ipynb
Jupyter Notebook
Phase - 1/Session4/1_Intializers.ipynb
yash1996/DL_curated_intuition
ae6d966bea93a0ece50dd7a2aa316ccde8c6d6b2
CC0-1.0
Phase - 1/Session4/1_Intializers.ipynb
yash1996/DL_curated_intuition
ae6d966bea93a0ece50dd7a2aa316ccde8c6d6b2
CC0-1.0
2020-09-26T00:39:54.000Z
2022-03-12T00:14:11.000Z
Phase - 1/Session4/1_Intializers.ipynb
nikhilpradeep/deep
d55562100fe3804e55ea1cf1637a669da69baec3
CC0-1.0
Qwen/Qwen-72B
1. YES
2. YES