neuron: $$\label{standard-initialization}
U \left[ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}} \right]$$ where $n$ is the number of neurons in the preceding layer (we call it [*standard initialization*]{} here). To the best of the authors’ knowledge, it is a heuristics.

In contrast, the following two weight initialization methods have been derived theoretically. Glorot and Bengio (2010) proposed the [*normalized initialization*]{}: the initial value of each weight between the layer $i$ and the layer $i+1$ is set according to the uniform distribution $$\label{glorot-initialization}
U \left[ -\frac{ \sqrt{6} }{ \sqrt{n_i + n_{i+1} }}, 
             \frac{ \sqrt{6} }{ \sqrt{n_i + n_{i+1} }} \right]$$ independently where $n_i$ is the number of neurons in the layer $i$. Eq. (\[glorot-initialization\]) was derived by keeping the variance of the output vector in each layer equal and keeping the variance of the back-propagated gradients equal, respectively, for the purpose of avoiding their saturation under the assumption that the variances of the input signals are all the same. The object of the normalized initialization is neural networks with symmetric activation function $f$ such that $\partial f(x)/\partial x \vert_{x=0} = 1$ (for example, sigmoid function and $\tanh$ function). Thus, the ReLU activation function is out of scope.

He et al. (2015) proposed a weight initialization method 