{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20843","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20843/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20843/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20843/events","html_url":"https://github.com/tensorflow/tensorflow/issues/20843","id":341575533,"node_id":"MDU6SXNzdWUzNDE1NzU1MzM=","number":20843,"title":"Performance decrease in TensorFlow 1.9 for large graphs","user":{"login":"drasmuss","id":1952220,"node_id":"MDQ6VXNlcjE5NTIyMjA=","avatar_url":"https://avatars1.githubusercontent.com/u/1952220?v=4","gravatar_id":"","url":"https://api.github.com/users/drasmuss","html_url":"https://github.com/drasmuss","followers_url":"https://api.github.com/users/drasmuss/followers","following_url":"https://api.github.com/users/drasmuss/following{/other_user}","gists_url":"https://api.github.com/users/drasmuss/gists{/gist_id}","starred_url":"https://api.github.com/users/drasmuss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drasmuss/subscriptions","organizations_url":"https://api.github.com/users/drasmuss/orgs","repos_url":"https://api.github.com/users/drasmuss/repos","events_url":"https://api.github.com/users/drasmuss/events{/privacy}","received_events_url":"https://api.github.com/users/drasmuss/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"tatatodd","id":5453737,"node_id":"MDQ6VXNlcjU0NTM3Mzc=","avatar_url":"https://avatars3.githubusercontent.com/u/5453737?v=4","gravatar_id":"","url":"https://api.github.com/users/tatatodd","html_url":"https://github.com/tatatodd","followers_url":"https://api.github.com/users/tatatodd/followers","following_url":"https://api.github.com/users/tatatodd/following{/other_user}","gists_url":"https://api.github.com/users/tatatodd/gists{/gist_id}","starred_url":"https://api.github.com/users/tatatodd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tatatodd/subscriptions","organizations_url":"https://api.github.com/users/tatatodd/orgs","repos_url":"https://api.github.com/users/tatatodd/repos","events_url":"https://api.github.com/users/tatatodd/events{/privacy}","received_events_url":"https://api.github.com/users/tatatodd/received_events","type":"User","site_admin":false},"assignees":[{"login":"tatatodd","id":5453737,"node_id":"MDQ6VXNlcjU0NTM3Mzc=","avatar_url":"https://avatars3.githubusercontent.com/u/5453737?v=4","gravatar_id":"","url":"https://api.github.com/users/tatatodd","html_url":"https://github.com/tatatodd","followers_url":"https://api.github.com/users/tatatodd/followers","following_url":"https://api.github.com/users/tatatodd/following{/other_user}","gists_url":"https://api.github.com/users/tatatodd/gists{/gist_id}","starred_url":"https://api.github.com/users/tatatodd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tatatodd/subscriptions","organizations_url":"https://api.github.com/users/tatatodd/orgs","repos_url":"https://api.github.com/users/tatatodd/repos","events_url":"https://api.github.com/users/tatatodd/events{/privacy}","received_events_url":"https://api.github.com/users/tatatodd/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-07-16T15:50:00Z","updated_at":"2018-08-15T14:46:21Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: yes\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Windows 10\r\n- **TensorFlow installed from (source or binary)**: binary\r\n- **TensorFlow version (use command below)**: 1.9.0/1.8.0\r\n- **Python version**: 3.5\r\n- **Bazel version (if compiling from source)**: n/a\r\n- **GCC/Compiler version (if compiling from source)**: n/a\r\n- **CUDA/cuDNN version**: 9.0\r\n- **GPU model and memory**: GTX 980 Ti, 6GB\r\n- **Exact command to reproduce**:\r\n\r\n```\r\nimport time\r\n\r\nimport tensorflow as tf\r\n\r\nx = tf.zeros((1, 10))\r\n\r\nfor i in range(5000):\r\n    x = tf.layers.dense(x, units=10)\r\n\r\nwith tf.Session() as sess:\r\n    sess.run(tf.global_variables_initializer())\r\n\r\n    print(\"a\")\r\n    start = time.time()\r\n    sess.run(x)\r\n    print(time.time() - start)\r\n\r\n    print(\"b\")\r\n    start = time.time()\r\n    sess.run(x)\r\n    print(time.time() - start)\r\n```\r\n\r\n### Describe the problem\r\nOn TensorFlow 1.9, this shows:\r\n```\r\na\r\n17.57940149307251\r\nb\r\n0.09075760841369629\r\n```\r\nOn TensorFlow 1.8, this shows:\r\n```\r\na\r\n11.689541101455688\r\nb\r\n0.08776521682739258\r\n```\r\nSo roughly a 50% increase in run time in TensorFlow 1.9.  Note that this only occurs on the first call to `sess.run` (\"a\" rather than \"b\"), so I suspect this is some change in the graph optimization step.  So, a couple questions:\r\n\r\n1. What changed?  I can't find anything obviously related to this in the release notes.\r\n2. Is this intended/expected (e.g., some new graph optimization steps were added that increase the initial time, but provide benefits in the long run)?  Or is it just a bug?\r\n3. If it is intended, is there some way to optionally disable this new feature if we find that initial build time is dominating the run time?\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/413220775","html_url":"https://github.com/tensorflow/tensorflow/issues/20843#issuecomment-413220775","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20843","id":413220775,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMzIyMDc3NQ==","user":{"login":"mihi-r","id":37024793,"node_id":"MDQ6VXNlcjM3MDI0Nzkz","avatar_url":"https://avatars1.githubusercontent.com/u/37024793?v=4","gravatar_id":"","url":"https://api.github.com/users/mihi-r","html_url":"https://github.com/mihi-r","followers_url":"https://api.github.com/users/mihi-r/followers","following_url":"https://api.github.com/users/mihi-r/following{/other_user}","gists_url":"https://api.github.com/users/mihi-r/gists{/gist_id}","starred_url":"https://api.github.com/users/mihi-r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mihi-r/subscriptions","organizations_url":"https://api.github.com/users/mihi-r/orgs","repos_url":"https://api.github.com/users/mihi-r/repos","events_url":"https://api.github.com/users/mihi-r/events{/privacy}","received_events_url":"https://api.github.com/users/mihi-r/received_events","type":"User","site_admin":false},"created_at":"2018-08-15T14:46:21Z","updated_at":"2018-08-15T14:46:21Z","author_association":"NONE","body":"I'm experiencing a fairly large slowdown too (on Windows 7 and Python 3.6; CPU) with the release of TensorFlow 1.9 and 1.10. For example, in 1.8 and versions prior, each evaluation step for the [CNN MNIST example model](https://www.tensorflow.org/tutorials/estimators/cnn) takes an average of 13 seconds while in 1.9 and 1.10, it trains for an average of 380 seconds. I hope this is still getting looked over."}}
{"actor":{"display_login":"mihi-r"}}