{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4033","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4033/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4033/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4033/events","html_url":"https://github.com/tensorflow/tensorflow/issues/4033","id":173117674,"number":4033,"title":"tensorflow.python.framework.errors.InvalidArgumentError: WhereOp: Race condition between counting the number of true elements and writing them","user":{"login":"abhijayghildyal","id":11029633,"avatar_url":"https://avatars.githubusercontent.com/u/11029633?v=3","gravatar_id":"","url":"https://api.github.com/users/abhijayghildyal","html_url":"https://github.com/abhijayghildyal","followers_url":"https://api.github.com/users/abhijayghildyal/followers","following_url":"https://api.github.com/users/abhijayghildyal/following{/other_user}","gists_url":"https://api.github.com/users/abhijayghildyal/gists{/gist_id}","starred_url":"https://api.github.com/users/abhijayghildyal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abhijayghildyal/subscriptions","organizations_url":"https://api.github.com/users/abhijayghildyal/orgs","repos_url":"https://api.github.com/users/abhijayghildyal/repos","events_url":"https://api.github.com/users/abhijayghildyal/events{/privacy}","received_events_url":"https://api.github.com/users/abhijayghildyal/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"benoitsteiner","id":6969686,"avatar_url":"https://avatars.githubusercontent.com/u/6969686?v=3","gravatar_id":"","url":"https://api.github.com/users/benoitsteiner","html_url":"https://github.com/benoitsteiner","followers_url":"https://api.github.com/users/benoitsteiner/followers","following_url":"https://api.github.com/users/benoitsteiner/following{/other_user}","gists_url":"https://api.github.com/users/benoitsteiner/gists{/gist_id}","starred_url":"https://api.github.com/users/benoitsteiner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benoitsteiner/subscriptions","organizations_url":"https://api.github.com/users/benoitsteiner/orgs","repos_url":"https://api.github.com/users/benoitsteiner/repos","events_url":"https://api.github.com/users/benoitsteiner/events{/privacy}","received_events_url":"https://api.github.com/users/benoitsteiner/received_events","type":"User","site_admin":false},"assignees":[{"login":"benoitsteiner","id":6969686,"avatar_url":"https://avatars.githubusercontent.com/u/6969686?v=3","gravatar_id":"","url":"https://api.github.com/users/benoitsteiner","html_url":"https://github.com/benoitsteiner","followers_url":"https://api.github.com/users/benoitsteiner/followers","following_url":"https://api.github.com/users/benoitsteiner/following{/other_user}","gists_url":"https://api.github.com/users/benoitsteiner/gists{/gist_id}","starred_url":"https://api.github.com/users/benoitsteiner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benoitsteiner/subscriptions","organizations_url":"https://api.github.com/users/benoitsteiner/orgs","repos_url":"https://api.github.com/users/benoitsteiner/repos","events_url":"https://api.github.com/users/benoitsteiner/events{/privacy}","received_events_url":"https://api.github.com/users/benoitsteiner/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2016-08-25T05:33:37Z","updated_at":"2016-08-29T15:45:27Z","closed_at":null,"body":"I am running `distributed tensorflow` version of `deep mnist` [(link)](https://www.tensorflow.org/versions/r0.10/tutorials/mnist/pros/index.html) (`asynchronous` data parallelism on different machines with gpu). The batch_size I have taken for one epoch is 1000 images and I am running 1000 epochs. My architecture has one `ps` on machine 1 and two workers, with `worker task_index=0` as `chief` on machine 2 and machine 1 has `worker task_index=1`.\r\n\r\nI really don't understand why I get this error sometimes when I run `worker task_index=1` on machine 1. The error does not persist though if I try running it a couple of times.\r\n\r\n    I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so.7.5 locally\r\n\tI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so.5 locally\r\n\tI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so.7.5 locally\r\n\tI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally\r\n\tI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so.7.5 locally\r\n\tcannot import name hashtable\r\n\tcannot import name hashtable\r\n\tcannot import name hashtable\r\n\tI tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n\tI tensorflow/core/common_runtime/gpu/gpu_init.cc:118] Found device 0 with properties: \r\n\tname: GeForce GTX 750 Ti\r\n\tmajor: 5 minor: 0 memoryClockRate (GHz) 1.124\r\n\tpciBusID 0000:02:00.0\r\n\tTotal memory: 2.00GiB\r\n\tFree memory: 125.53MiB\r\n\tI tensorflow/core/common_runtime/gpu/gpu_init.cc:138] DMA: 0 \r\n\tI tensorflow/core/common_runtime/gpu/gpu_init.cc:148] 0:   Y \r\n\tI tensorflow/core/common_runtime/gpu/gpu_device.cc:868] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:02:00.0)\r\n\tE tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to allocate 125.53M (131629056 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY\r\n\tI tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:206] Initialize HostPortsGrpcChannelCache for job ps -> {172.25.1.127:2223}\r\n\tI tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:206] Initialize HostPortsGrpcChannelCache for job worker -> {172.25.1.108:2222, localhost:2222}\r\n\tI tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:203] Started server with target: grpc://localhost:2222\r\n\tExtracting MNIST_data/train-images-idx3-ubyte.gz\r\n\tExtracting MNIST_data/train-labels-idx1-ubyte.gz\r\n\tExtracting MNIST_data/t10k-images-idx3-ubyte.gz\r\n\tExtracting MNIST_data/t10k-labels-idx1-ubyte.gz\r\n\tTraceback (most recent call last):\r\n\t  File \"dist_trainer_deepMnist_v2.py\", line 205, in <module>\r\n\t    tf.app.run()\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py\", line 30, in run\r\n\t    sys.exit(main(sys.argv))\r\n\t  File \"dist_trainer_deepMnist_v2.py\", line 157, in main\r\n\t    with sv.managed_session(server.target) as sess:\r\n\t  File \"/usr/lib/python2.7/contextlib.py\", line 17, in __enter__\r\n\t    return self.gen.next()\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py\", line 969, in managed_session\r\n\t    self.stop(close_summary_writer=close_summary_writer)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py\", line 797, in stop\r\n\t    stop_grace_period_secs=self._stop_grace_secs)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/coordinator.py\", line 386, in join\r\n\t    six.reraise(*self._exc_info_to_raise)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py\", line 958, in managed_session\r\n\t    start_standard_services=start_standard_services)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py\", line 722, in prepare_or_wait_for_session\r\n\t    max_wait_secs=max_wait_secs)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py\", line 351, in wait_for_session\r\n\t    is_ready, not_ready_msg = self._model_ready(sess)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py\", line 437, in _model_ready\r\n\t    return self._ready(self._ready_op, sess, \"Model not ready\")\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py\", line 406, in _ready\r\n\t    ready_value = sess.run(op)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 710, in run\r\n\t    run_metadata_ptr)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 908, in _run\r\n\t    feed_dict_string, options, run_metadata)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 958, in _do_run\r\n\t    target_list, options, run_metadata)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 978, in _do_call\r\n\t    raise type(e)(node_def, op, message)\r\n\ttensorflow.python.framework.errors.InvalidArgumentError: WhereOp: Race condition between counting the number of true elements and writing them.  When counting, saw 2402 elements; but when writing their indices, saw 19 elements.\r\n\t\t [[Node: report_uninitialized_variables/Where = Where[_device=\"/job:ps/replica:0/task:0/cpu:0\"](report_uninitialized_variables/Reshape_1)]]\r\n\tCaused by op u'report_uninitialized_variables/Where', defined at:\r\n\t  File \"dist_trainer_deepMnist_v2.py\", line 205, in <module>\r\n\t    tf.app.run()\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py\", line 30, in run\r\n\t    sys.exit(main(sys.argv))\r\n\t  File \"dist_trainer_deepMnist_v2.py\", line 153, in main\r\n\t    save_model_secs=600)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py\", line 310, in __init__\r\n\t    ready_op=ready_op, ready_for_local_init_op=ready_for_local_init_op)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py\", line 399, in _init_ready_op\r\n\t    ready_op = variables.report_uninitialized_variables()\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py\", line 1031, in report_uninitialized_variables\r\n\t    name=name)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py\", line 888, in boolean_mask\r\n\t    return _apply_mask_1d(tensor, mask)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py\", line 863, in _apply_mask_1d\r\n\t    indices = squeeze(where(mask), squeeze_dims=[1])\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py\", line 2663, in where\r\n\t    result = _op_def_lib.apply_op(\"Where\", input=input, name=name)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py\", line 703, in apply_op\r\n\t    op_def=op_def)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py\", line 2333, in create_op\r\n\t    original_op=self._default_original_op, op_def=op_def)\r\n\t  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py\", line 1252, in __init__\r\n\t    self._traceback = _extract_stack()\r\n\r\nI tried looking for a reason as to why this error occurs, but couldn't find one. One clue is this part in the error\r\n\r\n    tensorflow.python.framework.errors.InvalidArgumentError: WhereOp: Race condition between counting the number of true elements and writing them.  When counting, saw 2402 elements; but when writing their indices, saw 19 elements.\r\n     [[Node: report_uninitialized_variables/Where = Where[_device=\"/job:ps/replica:0/task:0/cpu:0\"](report_uninitialized_variables/Reshape_1)]]\r\n    Caused by op u'report_uninitialized_variables/Where',\r\n\r\nAnother clue is \r\n\r\n    E tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to allocate 125.53M (131629056 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY\r\n\r\nthis is there when I am running the ps on the same machine which is also using the gpu. It'll be great if someone can explain why this happens?\r\n\r\n(Asked the same on [stack overflow](http://stackoverflow.com/questions/39130553/tensorflow-python-framework-errors-invalidargumenterror-whereop-race-condition?noredirect=1#comment65613229_39130553))"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/243163770","html_url":"https://github.com/tensorflow/tensorflow/issues/4033#issuecomment-243163770","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4033","id":243163770,"user":{"login":"prb12","id":11547801,"avatar_url":"https://avatars.githubusercontent.com/u/11547801?v=3","gravatar_id":"","url":"https://api.github.com/users/prb12","html_url":"https://github.com/prb12","followers_url":"https://api.github.com/users/prb12/followers","following_url":"https://api.github.com/users/prb12/following{/other_user}","gists_url":"https://api.github.com/users/prb12/gists{/gist_id}","starred_url":"https://api.github.com/users/prb12/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prb12/subscriptions","organizations_url":"https://api.github.com/users/prb12/orgs","repos_url":"https://api.github.com/users/prb12/repos","events_url":"https://api.github.com/users/prb12/events{/privacy}","received_events_url":"https://api.github.com/users/prb12/received_events","type":"User","site_admin":false},"created_at":"2016-08-29T15:45:27Z","updated_at":"2016-08-29T15:45:27Z","body":"> When i run the three tasks on three different machines, the error does not occur.\r\n\r\n@mrry  I've now seen two issues where strange things happen when we are running more than one TF task on the same machine.  I can't think of any way this might affect execution other than timing?"}}
{"actor":{"display_login":"prb12"}}