{"action":"closed","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/18706","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/18706/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/18706/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/18706/events","html_url":"https://github.com/tensorflow/tensorflow/issues/18706","id":316038826,"node_id":"MDU6SXNzdWUzMTYwMzg4MjY=","number":18706,"title":"Wrong variance from initialisers","user":{"login":"mrTsjolder","id":3738082,"node_id":"MDQ6VXNlcjM3MzgwODI=","avatar_url":"https://avatars2.githubusercontent.com/u/3738082?v=4","gravatar_id":"","url":"https://api.github.com/users/mrTsjolder","html_url":"https://github.com/mrTsjolder","followers_url":"https://api.github.com/users/mrTsjolder/followers","following_url":"https://api.github.com/users/mrTsjolder/following{/other_user}","gists_url":"https://api.github.com/users/mrTsjolder/gists{/gist_id}","starred_url":"https://api.github.com/users/mrTsjolder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrTsjolder/subscriptions","organizations_url":"https://api.github.com/users/mrTsjolder/orgs","repos_url":"https://api.github.com/users/mrTsjolder/repos","events_url":"https://api.github.com/users/mrTsjolder/events{/privacy}","received_events_url":"https://api.github.com/users/mrTsjolder/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"shivaniag","id":16565716,"node_id":"MDQ6VXNlcjE2NTY1NzE2","avatar_url":"https://avatars1.githubusercontent.com/u/16565716?v=4","gravatar_id":"","url":"https://api.github.com/users/shivaniag","html_url":"https://github.com/shivaniag","followers_url":"https://api.github.com/users/shivaniag/followers","following_url":"https://api.github.com/users/shivaniag/following{/other_user}","gists_url":"https://api.github.com/users/shivaniag/gists{/gist_id}","starred_url":"https://api.github.com/users/shivaniag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shivaniag/subscriptions","organizations_url":"https://api.github.com/users/shivaniag/orgs","repos_url":"https://api.github.com/users/shivaniag/repos","events_url":"https://api.github.com/users/shivaniag/events{/privacy}","received_events_url":"https://api.github.com/users/shivaniag/received_events","type":"User","site_admin":false},"assignees":[{"login":"shivaniag","id":16565716,"node_id":"MDQ6VXNlcjE2NTY1NzE2","avatar_url":"https://avatars1.githubusercontent.com/u/16565716?v=4","gravatar_id":"","url":"https://api.github.com/users/shivaniag","html_url":"https://github.com/shivaniag","followers_url":"https://api.github.com/users/shivaniag/followers","following_url":"https://api.github.com/users/shivaniag/following{/other_user}","gists_url":"https://api.github.com/users/shivaniag/gists{/gist_id}","starred_url":"https://api.github.com/users/shivaniag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shivaniag/subscriptions","organizations_url":"https://api.github.com/users/shivaniag/orgs","repos_url":"https://api.github.com/users/shivaniag/repos","events_url":"https://api.github.com/users/shivaniag/events{/privacy}","received_events_url":"https://api.github.com/users/shivaniag/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2018-04-19T21:00:02Z","updated_at":"2018-06-04T20:00:28Z","closed_at":"2018-06-04T20:00:28Z","author_association":"CONTRIBUTOR","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: no\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: N/A\r\n- **TensorFlow installed from (source or binary)**: N/A\r\n- **TensorFlow version (use command below)**: 1.3, 1.4, 1.5, 1.6, 1.7, 1.8\r\n- **Python version**: N/A\r\n- **Bazel version (if compiling from source)**: N/A\r\n- **GCC/Compiler version (if compiling from source)**: N/A\r\n- **CUDA/cuDNN version**: N/A\r\n- **GPU model and memory**: N/A\r\n- **Exact command to reproduce**: See *Source Code* section\r\n\r\n\r\n### Describe the problem\r\nProbably the best known fact about weight initialisation is that initial weights should have small values, close to zero. Probably, a lot of people, especially those that read [1] by LeCun et al., also know that also some properties of the distribution from which the initial values are drawn matters. This has been discussed in great detail in the literature. Xavier Glorot argued in [2] that the variance of the (initial) weights should have zero mean and a variance of 1/(fan_in + fan_out) for sigmoid activation functions (focusing on tanh). He et al. [3] noted that  for ReLUs, the weights should have zero mean and a variance of 2/fan_in in order to correct for the variance that gets lost in the negative part. Finally, the entire theory of self-normalizing networks, introduced in [4] by Klambauer et al., and the parameters for alpha and gamma for SELU activation functions, is built upon the assumptions that the weights have mean 0 and variance 1/fan_in.\r\n\r\nThe [truncated normal distribution](https://en.wikipedia.org/wiki/Truncated_normal_distribution) is a commonly used distribution for initialising weights in a neural network. After all, it looks like a Gaussian, but does not allow outliers, which could cause saturation for certain activation function from the start. It also has a mean and variance that depend on the four parameters of the truncated normal distribution: mean, standard deviation, lower boundary and upper boundary. This distribution is implemented in `tf.truncated_normal` and allows to create a truncated normal distribution with arguments for its mean and standard deviation, i.e. the standard distribution of the Gaussian before truncating (I know that this is confusing #13686). Note that the boundaries are fixed to be at 2 standard deviations at each side of the mean.\r\n\r\nThis is all fine, until one wants to create a truncated normal distribution with a specific standard deviation, which happens to be the case for the initialisers. Although the well-studied initialisers, mentioned earlier, tell us exactly what the best variances are, the `<name>_normal_initializer` lead to wrong initialisations (for examples, see below). The problem can be brought back to the `VarianceScaling` initialiser, which allows to choose between a normal and uniform distribution. When using `distribution='normal'`, a truncated normal is used under the hood. Because there is no compensation for the truncation of the normal distribution (which would have had the correct standard deviation), the variance of the generated samples do not have the required variance. This has been resolved in Theano from the start (see issue theano/theano#6381) and I wanted to tackle this problem in Keras (see issue keras-team/keras#8048), where I was redirected here (keras-team/keras#9963).\r\n\r\nIt should be possible to map the solution implemented in keras-team/keras#9963 directly to the tensorflow code. I hope this gets fixed soon, because there are probably plenty of people using wrong initialisations due to this faulty implementation.\r\n\r\nPS: I am not eager to accept all these terms to issue a pull request for making this minor contribution directly. Thanks to anybody who would take this fix upon him/her.\r\n\r\nTL;DR; The default initialisers using truncated normal distributions are simply wrong. By truncating a normal distribution, variance gets lost and the entire theoretical foundations of the initialisers disappear. I believe that this issue can be fixed by accounting for the truncation in the `VarianceScaling` initialiser.\r\n\r\n### Source code / logs\r\n\r\n    import tensorflow as tf\r\n    import numpy as np\r\n\r\n    fan_in, fan_out = 1000, 100\r\n\r\n    uniform_init = tf.variance_scaling_initializer(distribution='uniform')\r\n    normal_init = tf.variance_scaling_initializer(distribution='normal')\r\n\r\n    print(\"expected variance: {:f}\".format(1. / fan_in))\r\n    with tf.Session() as sess:\r\n         print(\" uniform variance: {:f}\".format(\r\n             np.var(sess.run(uniform_init((fan_in, fan_out))))))\r\n         print(\"  normal variance: {:f}\".format(\r\n             np.std(sess.run(normal_init((fan_in, fan_out))))))\r\n\r\n    uniform_init = tf.glorot_uniform_initializer()\r\n    normal_init = tf.glorot_normal_initializer()\r\n\r\n    print(\"expected variance: {:f}\".format(2. / (fan_in + fan_out)))\r\n    with tf.Session() as sess:\r\n         print(\" uniform variance: {:f}\".format(\r\n             np.var(sess.run(uniform_init((fan_in, fan_out))))))\r\n         print(\"  normal variance: {:f}\".format(\r\n             np.std(sess.run(normal_init((fan_in, fan_out))))))\r\n\r\n### References\r\n[1] LeCun, Yann, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. \"Efficient backprop.\" In Neural networks: Tricks of the trade, pp. 9-50. Springer, Berlin, Heidelberg, 1998.\r\n[2] Glorot, Xavier, and Yoshua Bengio. \"Understanding the difficulty of training deep feedforward neural networks.\" In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. 2010.\r\n[3] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. \"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.\" In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034. 2015.\r\n[4] Klambauer, Günter, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. \"Self-normalizing neural networks.\" Advances in Neural Information Processing Systems (2017): 972-981.\r\n"}}
{"actor":{"display_login":"mrTsjolder"}}