networks have attracted attention extensively because of their human-like intelligence achieved through learning and generalization. To date, deep neural networks have produced outstanding results in the fields of image processing and speech recognition (Mohamed et al., 2009; Seide et al., 2011; Taigman et al., 2014). Moreover, their scope of application has expanded, for example, to the field of machine translation (Sutskever et al., 2014).

In using deep neural networks, finding a good initialization becomes extremely important to achieve good results. Heuristics have been used for weight initialization of neural networks for a long time. For example, a uniform distribution $
U [ -1/\sqrt{n}, 1/\sqrt{n} ]
$ has been often used where $n$ is the number of neurons in the preceding layer. Pre-training might be regarded as a kind of weight initialization methods, which could avoid local minima and plateaus (Bengio et al., 2007). However, several theoretical researches on weight initialization methods have been progressing in recent years. Glorot and Bengio (2010) derived a theoretically sound uniform distribution $
U [\ - \sqrt{6} / \sqrt{n_i + n_{i+1} }, 
          \sqrt{6} /\sqrt{n_i + n_{i+1} } \ ] 
$ for the weight initialization of deep neural networks with an activation function which is symmetric 