{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16889","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16889/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16889/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16889/events","html_url":"https://github.com/tensorflow/tensorflow/issues/16889","id":295856975,"number":16889,"title":"Problems Getting TensorFlow to behave Deterministically","user":{"login":"rundembear","id":7693798,"avatar_url":"https://avatars0.githubusercontent.com/u/7693798?v=4","gravatar_id":"","url":"https://api.github.com/users/rundembear","html_url":"https://github.com/rundembear","followers_url":"https://api.github.com/users/rundembear/followers","following_url":"https://api.github.com/users/rundembear/following{/other_user}","gists_url":"https://api.github.com/users/rundembear/gists{/gist_id}","starred_url":"https://api.github.com/users/rundembear/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rundembear/subscriptions","organizations_url":"https://api.github.com/users/rundembear/orgs","repos_url":"https://api.github.com/users/rundembear/repos","events_url":"https://api.github.com/users/rundembear/events{/privacy}","received_events_url":"https://api.github.com/users/rundembear/received_events","type":"User","site_admin":false},"labels":[{"id":386191887,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20response","name":"stat:awaiting response","color":"5319e7","default":false},{"id":473172988,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:bug/performance","name":"type:bug/performance","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":{"login":"shivaniag","id":16565716,"avatar_url":"https://avatars1.githubusercontent.com/u/16565716?v=4","gravatar_id":"","url":"https://api.github.com/users/shivaniag","html_url":"https://github.com/shivaniag","followers_url":"https://api.github.com/users/shivaniag/followers","following_url":"https://api.github.com/users/shivaniag/following{/other_user}","gists_url":"https://api.github.com/users/shivaniag/gists{/gist_id}","starred_url":"https://api.github.com/users/shivaniag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shivaniag/subscriptions","organizations_url":"https://api.github.com/users/shivaniag/orgs","repos_url":"https://api.github.com/users/shivaniag/repos","events_url":"https://api.github.com/users/shivaniag/events{/privacy}","received_events_url":"https://api.github.com/users/shivaniag/received_events","type":"User","site_admin":false},"assignees":[{"login":"shivaniag","id":16565716,"avatar_url":"https://avatars1.githubusercontent.com/u/16565716?v=4","gravatar_id":"","url":"https://api.github.com/users/shivaniag","html_url":"https://github.com/shivaniag","followers_url":"https://api.github.com/users/shivaniag/followers","following_url":"https://api.github.com/users/shivaniag/following{/other_user}","gists_url":"https://api.github.com/users/shivaniag/gists{/gist_id}","starred_url":"https://api.github.com/users/shivaniag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shivaniag/subscriptions","organizations_url":"https://api.github.com/users/shivaniag/orgs","repos_url":"https://api.github.com/users/shivaniag/repos","events_url":"https://api.github.com/users/shivaniag/events{/privacy}","received_events_url":"https://api.github.com/users/shivaniag/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2018-02-09T12:36:18Z","updated_at":"2018-03-09T01:53:34Z","closed_at":null,"author_association":"NONE","body":"Please go to Stack Overflow for help and support:\r\n\r\nhttps://stackoverflow.com/questions/tagged/tensorflow\r\n\r\nIf you open a GitHub issue, here is our policy:\r\n\r\n1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).\r\n2. The form below must be filled out.\r\n3. It shouldn't be a TensorBoard issue. Those go [here](https://github.com/tensorflow/tensorboard/issues).\r\n\r\n**Here's why we have that policy**: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.\r\n\r\n------------------------\r\n\r\n### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**:\r\n\r\nYes, the code I am working with is company proprietary. If you need an example I will need to try to extract something that I can share. \r\n\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:\r\n\r\nMac OS 10.13.2\r\n\r\n- **TensorFlow installed from (source or binary)**:\r\n\r\nbinary\r\n\r\n- **TensorFlow version (use command below)**:\r\n\r\nv1.3.0-rc2-20-g0787eee 1.3.0\r\n\r\n- **Python version**: \r\n\r\n3.6.4\r\n\r\n- **Bazel version (if compiling from source)**:\r\n- **GCC/Compiler version (if compiling from source)**:\r\n- **CUDA/cuDNN version**:\r\n\r\nRunning on CPU.\r\n\r\n- **GPU model and memory**:\r\n\r\n- **Exact command to reproduce**:\r\n\r\nPlease see description of problem. I don't currently have an example I can give you but I am hoping that the information I am providing is enough to get at least some idea about what is going on.\r\n\r\nYou can collect some of this information using our environment capture script:\r\n\r\nhttps://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh\r\n\r\nYou can obtain the TensorFlow version with\r\n\r\npython -c \"import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)\"\r\n\r\n### Describe the problem\r\nDescribe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.\r\n\r\nI am having a major struggle trying to get TensorFlow to behave in a deterministic manner. I started out trying to compare two different versions of my source code, but the numeric differences from one execution to the next were large enough to make comparisons problematic.\r\n\r\nSo I went to trying to run exactly the same code in two different shells. And this is where things got weird.\r\n\r\nI went through the code, and everywhere where TensorFlow is imported, everywhere I did operations using the form \"with graph as default:\", and prior to each call to run I added in a call to     tf.set_random_seed(42) [and yes, I used 42 in all of the calls].\r\n\r\nI also placed breakpoints before anything in my code which was a tf call with 'random' in the name (I even searched the docs to see if there were random calls that didn't have random in the name, I found a couple but I am not using them). In all of the random calls that I am using I added in a seed=42 argument. So as far as I can tell I have covered all of the bases.\r\n\r\nSo what happens? I run the code in one shell, get a set of numbers. Run code in a second shell, get identical numbers. Good. Run the code a few more times, keep getting the same numbers. Better.\r\nRun the code again, get completely different numbers!!! If I keep running the code in the different shells sometimes I get the same numbers from execution to execution. Sometimes I get different numbers. Sometimes it goes on to completely new numbers. Sometimes it goes back to an older set of numbers.\r\n\r\nSo I am getting something that is consistent enough that it feels like I haven't missed any places where I need to set the seed, but not consistent enough to say that it is deterministic.\r\n\r\nI really want to be able to run the code over and over again and get exactly the same numeric result. Is this an unreasonable expectation? \r\n\r\nCan anyone offer an explanation as to why the code would generate exactly the same result (and I'm looking down to 8 decimal places so this is fairly precise) five times in a row, and then switch to a completely different result and then repeat that result multiple times in a row?\r\n\r\nI can certainly try to reduce this into an example I can share (the model I am using is a toy model we use for unit testing). I wanted to see if anyone had any insight into the general problem before I put the time into that.\r\n\r\n\r\n\r\n\r\n### Source code / logs\r\nInclude any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.\r\n\r\nI will include two examples of the numeric output I am getting. This data is part of the returned result from Session.run().\r\n\r\nExample A:\r\n\r\nout_node_weights [[ 9.09113979  9.09934711  9.09535408  9.09768963  9.11262989  9.09756184\r\n   9.10322666  9.09527016  9.10811996  9.09029293]\r\n [ 9.34178257  9.34915924  9.34216785  9.34740925  9.32617569  9.34794617\r\n   9.34459496  9.35203457  9.34202194  9.34494209]\r\n [ 9.09113979  9.09934711  9.09535408  9.09768963  9.11262989  9.09756184\r\n   9.10322666  9.09527016  9.10811996  9.09029293]\r\n [ 9.09113979  9.09934711  9.09535408  9.09768963  9.11262989  9.09756184\r\n   9.10322666  9.09527016  9.10811996  9.09029293]\r\n [ 9.59115124  9.59841442  9.59205914  9.59892082  9.58935356  9.59801197\r\n   9.59655094  9.5970974   9.5989027   9.59136868]\r\n [ 9.39457417  9.39968395  9.39309788  9.4028101   9.40468502  9.39697838\r\n   9.39671326  9.39805126  9.39696026  9.39565849]]\r\n\r\nExample B:\r\n\r\nout_node_weights [[ 9.04592514  9.05451775  9.05065632  9.05263042  9.06681919  9.05208111\r\n   9.05834484  9.05059719  9.06315136  9.04525948]\r\n [ 9.25311852  9.26151276  9.25516319  9.25806046  9.23816681  9.26067734\r\n   9.25860214  9.26416397  9.25700569  9.25596619]\r\n [ 9.27453995  9.28063679  9.27472687  9.28252983  9.28667641  9.27761269\r\n   9.27913475  9.27859116  9.28033924  9.27534485]\r\n [ 9.27453995  9.28063679  9.27472687  9.28252983  9.28667641  9.27761269\r\n   9.27913475  9.27859116  9.28033924  9.27534485]\r\n [ 9.50198936  9.51004601  9.50419044  9.50922203  9.50067139  9.50993729\r\n   9.50946426  9.5085659   9.51255226  9.50196838]\r\n [ 9.27453995  9.28063679  9.27472687  9.28252983  9.28667641  9.27761269\r\n   9.27913475  9.27859116  9.28033924  9.27534485]]"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/371687327","html_url":"https://github.com/tensorflow/tensorflow/issues/16889#issuecomment-371687327","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16889","id":371687327,"user":{"login":"shivaniag","id":16565716,"avatar_url":"https://avatars1.githubusercontent.com/u/16565716?v=4","gravatar_id":"","url":"https://api.github.com/users/shivaniag","html_url":"https://github.com/shivaniag","followers_url":"https://api.github.com/users/shivaniag/followers","following_url":"https://api.github.com/users/shivaniag/following{/other_user}","gists_url":"https://api.github.com/users/shivaniag/gists{/gist_id}","starred_url":"https://api.github.com/users/shivaniag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shivaniag/subscriptions","organizations_url":"https://api.github.com/users/shivaniag/orgs","repos_url":"https://api.github.com/users/shivaniag/repos","events_url":"https://api.github.com/users/shivaniag/events{/privacy}","received_events_url":"https://api.github.com/users/shivaniag/received_events","type":"User","site_admin":false},"created_at":"2018-03-09T01:53:34Z","updated_at":"2018-03-09T01:53:34Z","author_association":"CONTRIBUTOR","body":"Do you mind sharing the code you are running here, it is really difficult to help without that information. Thanks!"}}
{"actor":{"display_login":"shivaniag"}}