{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21470","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21470/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21470/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21470/events","html_url":"https://github.com/tensorflow/tensorflow/issues/21470","id":348629006,"node_id":"MDU6SXNzdWUzNDg2MjkwMDY=","number":21470,"title":"Problem with tf.contrib.distribute.MirroredStrategy() in Windows OS","user":{"login":"Windaway","id":4530735,"node_id":"MDQ6VXNlcjQ1MzA3MzU=","avatar_url":"https://avatars0.githubusercontent.com/u/4530735?v=4","gravatar_id":"","url":"https://api.github.com/users/Windaway","html_url":"https://github.com/Windaway","followers_url":"https://api.github.com/users/Windaway/followers","following_url":"https://api.github.com/users/Windaway/following{/other_user}","gists_url":"https://api.github.com/users/Windaway/gists{/gist_id}","starred_url":"https://api.github.com/users/Windaway/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Windaway/subscriptions","organizations_url":"https://api.github.com/users/Windaway/orgs","repos_url":"https://api.github.com/users/Windaway/repos","events_url":"https://api.github.com/users/Windaway/events{/privacy}","received_events_url":"https://api.github.com/users/Windaway/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"yuefengz","id":1647833,"node_id":"MDQ6VXNlcjE2NDc4MzM=","avatar_url":"https://avatars0.githubusercontent.com/u/1647833?v=4","gravatar_id":"","url":"https://api.github.com/users/yuefengz","html_url":"https://github.com/yuefengz","followers_url":"https://api.github.com/users/yuefengz/followers","following_url":"https://api.github.com/users/yuefengz/following{/other_user}","gists_url":"https://api.github.com/users/yuefengz/gists{/gist_id}","starred_url":"https://api.github.com/users/yuefengz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yuefengz/subscriptions","organizations_url":"https://api.github.com/users/yuefengz/orgs","repos_url":"https://api.github.com/users/yuefengz/repos","events_url":"https://api.github.com/users/yuefengz/events{/privacy}","received_events_url":"https://api.github.com/users/yuefengz/received_events","type":"User","site_admin":false},"assignees":[{"login":"yuefengz","id":1647833,"node_id":"MDQ6VXNlcjE2NDc4MzM=","avatar_url":"https://avatars0.githubusercontent.com/u/1647833?v=4","gravatar_id":"","url":"https://api.github.com/users/yuefengz","html_url":"https://github.com/yuefengz","followers_url":"https://api.github.com/users/yuefengz/followers","following_url":"https://api.github.com/users/yuefengz/following{/other_user}","gists_url":"https://api.github.com/users/yuefengz/gists{/gist_id}","starred_url":"https://api.github.com/users/yuefengz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yuefengz/subscriptions","organizations_url":"https://api.github.com/users/yuefengz/orgs","repos_url":"https://api.github.com/users/yuefengz/repos","events_url":"https://api.github.com/users/yuefengz/events{/privacy}","received_events_url":"https://api.github.com/users/yuefengz/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-08-08T08:39:51Z","updated_at":"2018-08-31T15:03:07Z","closed_at":null,"author_association":"NONE","body":"------------------------\r\n### System information\r\nWindows 10 x64 pro 17314.365, i7 7900X R6E, TTxp (*4), 16G DDR4 3000@2666 (*6) \r\nChannel .\r\nTensorflow 1.9.0 with AVX2 simd, CUDA9.2.148, Cudnn7.1.4, py3.6.6 , VS2017 15.7(with cmake), CP/SM6.1\r\n\r\n### Describe the problem\r\nOp type not registered 'NcclAllReduce' in binary running on Windows10.\r\nWith onedevicestrategy no problems. I noticed that NCCL only works in Linux.\r\n\r\n### Source code / logs\r\ntensorflow/contrib/distribute/python/examples/simple_tfkeras_example.py\r\n\r\n\r\n> `\r\n> from __future__ import absolute_import\r\n> from __future__ import division\r\n> from __future__ import print_function\r\n> \r\n> import sys\r\n> \r\n> import numpy as np\r\n> import tensorflow as tf\r\n> \r\n> \r\n> def input_fn():\r\n>   x = np.random.random((1024, 10))\r\n>   y = np.random.randint(2, size=(1024, 1))\r\n>   x = tf.cast(x, tf.float32)\r\n>   dataset = tf.data.Dataset.from_tensor_slices((x, y))\r\n>   dataset = dataset.repeat(10)\r\n>   dataset = dataset.batch(32)\r\n>   return dataset\r\n> \r\n> \r\n> def main(args):\r\n> \r\n>   model_dir = './'\r\n>   print('Using %s to store checkpoints.' % model_dir)\r\n> \r\n>   # Define tf.keras Model.\r\n>   model = tf.keras.Sequential()\r\n>   model.add(tf.keras.layers.Dense(16, activation='relu', input_shape=(10,)))\r\n>   model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\r\n> \r\n>   # Compile tf.keras Model.\r\n>   optimizer = tf.train.GradientDescentOptimizer(0.2)\r\n>   model.compile(loss='binary_crossentropy', optimizer=optimizer)\r\n>   model.summary()\r\n>   tf.keras.backend.set_learning_phase(True)\r\n> \r\n>   # Define a DistributionStrategy and convert the tf.keras Model to a\r\n>   # tf.Estimator that utilizes the DistributionStrategy.\r\n>   strategy = tf.contrib.distribute.MirroredStrategy()\r\n>   config = tf.estimator.RunConfig(\r\n>       train_distribute=strategy)\r\n>   keras_estimator = tf.keras.estimator.model_to_estimator(\r\n>       keras_model=model, config=config, model_dir=model_dir)\r\n> \r\n>   # Train and evaluate the tf.Estimator.\r\n>   keras_estimator.train(input_fn=input_fn, steps=10)\r\n>   eval_result = keras_estimator.evaluate(input_fn=input_fn)\r\n>   print('Eval result: {}'.format(eval_result))\r\n> \r\n> if __name__ == '__main__':\r\n>   tf.app.run(argv=sys.argv)`\r\n\r\nLogs:\r\n\r\n\r\n> D:\\Anaconda3\\envs\\tensorflow\\python.exe Z:/PyProj/leaf/othertest.py\r\n> Using ./ to store checkpoints.\r\n> _________________________________________________________________\r\n> Layer (type)                 Output Shape              Param #   \r\n> =================================================================\r\n> dense (Dense)                (None, 16)                176       \r\n> _________________________________________________________________\r\n> dense_1 (Dense)              (None, 1)                 17        \r\n> =================================================================\r\n> Total params: 193\r\n> Trainable params: 193\r\n> Non-trainable params: 0\r\n> _________________________________________________________________\r\n> 2018-08-08 16:28:32.965695: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1392] Found device 0 with properties: \r\n> name: TITAN Xp COLLECTORS EDITION major: 6 minor: 1 memoryClockRate(GHz): 1.582\r\n> pciBusID: 0000:17:00.0\r\n> totalMemory: 12.00GiB freeMemory: 9.93GiB\r\n> 2018-08-08 16:28:33.068708: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1392] Found device 1 with properties: \r\n> name: TITAN Xp COLLECTORS EDITION major: 6 minor: 1 memoryClockRate(GHz): 1.582\r\n> pciBusID: 0000:18:00.0\r\n> totalMemory: 12.00GiB freeMemory: 9.93GiB\r\n> 2018-08-08 16:28:33.170068: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1392] Found device 2 with properties: \r\n> name: TITAN Xp COLLECTORS EDITION major: 6 minor: 1 memoryClockRate(GHz): 1.582\r\n> pciBusID: 0000:65:00.0\r\n> totalMemory: 12.00GiB freeMemory: 9.93GiB\r\n> 2018-08-08 16:28:33.280326: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1392] Found device 3 with properties: \r\n> name: TITAN Xp COLLECTORS EDITION major: 6 minor: 1 memoryClockRate(GHz): 1.582\r\n> pciBusID: 0000:b3:00.0\r\n> totalMemory: 12.00GiB freeMemory: 9.93GiB\r\n> 2018-08-08 16:28:33.281257: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1471] Adding visible gpu devices: 0, 1, 2, 3\r\n> 2018-08-08 16:28:35.779554: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n> 2018-08-08 16:28:35.779811: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:958]      0 1 2 3 \r\n> 2018-08-08 16:28:35.779990: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 0:   N N N N \r\n> 2018-08-08 16:28:35.780385: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 1:   N N N N \r\n> 2018-08-08 16:28:35.780639: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 2:   N N N N \r\n> 2018-08-08 16:28:35.783787: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 3:   N N N N \r\n> 2018-08-08 16:28:35.784232: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9612 MB memory) -> physical GPU (device: 0, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:17:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:36.220466: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9612 MB memory) -> physical GPU (device: 1, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:18:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:36.655982: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9612 MB memory) -> physical GPU (device: 2, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:65:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.091894: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 9612 MB memory) -> physical GPU (device: 3, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:b3:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.529496: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1471] Adding visible gpu devices: 0, 1, 2, 3\r\n> 2018-08-08 16:28:37.530168: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n> 2018-08-08 16:28:37.530377: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:958]      0 1 2 3 \r\n> 2018-08-08 16:28:37.530527: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 0:   N N N N \r\n> 2018-08-08 16:28:37.530675: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 1:   N N N N \r\n> 2018-08-08 16:28:37.530822: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 2:   N N N N \r\n> 2018-08-08 16:28:37.530971: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 3:   N N N N \r\n> 2018-08-08 16:28:37.531268: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9612 MB memory) -> physical GPU (device: 0, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:17:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.531988: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9612 MB memory) -> physical GPU (device: 1, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:18:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.532651: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9612 MB memory) -> physical GPU (device: 2, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:65:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.533329: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 9612 MB memory) -> physical GPU (device: 3, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:b3:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.570852: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1471] Adding visible gpu devices: 0, 1, 2, 3\r\n> 2018-08-08 16:28:37.571420: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n> 2018-08-08 16:28:37.571620: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:958]      0 1 2 3 \r\n> 2018-08-08 16:28:37.571766: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 0:   N N N N \r\n> 2018-08-08 16:28:37.571910: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 1:   N N N N \r\n> 2018-08-08 16:28:37.572056: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 2:   N N N N \r\n> 2018-08-08 16:28:37.572200: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 3:   N N N N \r\n> 2018-08-08 16:28:37.572479: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9612 MB memory) -> physical GPU (device: 0, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:17:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.573133: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9612 MB memory) -> physical GPU (device: 1, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:18:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.573824: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9612 MB memory) -> physical GPU (device: 2, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:65:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.574554: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 9612 MB memory) -> physical GPU (device: 3, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:b3:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.902576: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1471] Adding visible gpu devices: 0, 1, 2, 3\r\n> 2018-08-08 16:28:37.903108: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n> 2018-08-08 16:28:37.903311: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:958]      0 1 2 3 \r\n> 2018-08-08 16:28:37.903456: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 0:   N N N N \r\n> 2018-08-08 16:28:37.903604: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 1:   N N N N \r\n> 2018-08-08 16:28:37.903751: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 2:   N N N N \r\n> 2018-08-08 16:28:37.903896: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:971] 3:   N N N N \r\n> 2018-08-08 16:28:37.904189: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/device:GPU:0 with 9612 MB memory) -> physical GPU (device: 0, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:17:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.904886: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/device:GPU:1 with 9612 MB memory) -> physical GPU (device: 1, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:18:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.905868: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/device:GPU:2 with 9612 MB memory) -> physical GPU (device: 2, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:65:00.0, compute capability: 6.1)\r\n> 2018-08-08 16:28:37.906462: I c:\\users\\user\\source\\repos\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1084] Created TensorFlow device (/device:GPU:3 with 9612 MB memory) -> physical GPU (device: 3, name: TITAN Xp COLLECTORS EDITION, pci bus id: 0000:b3:00.0, compute capability: 6.1)\r\n> Traceback (most recent call last):\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 1589, in _create_c_op\r\n>     c_op = c_api.TF_FinishOperation(op_desc)\r\n> tensorflow.python.framework.errors_impl.InvalidArgumentError: Op type not registered 'NcclAllReduce' in binary running on DESKTOP-K1BEKCL. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed. while building NodeDef 'NcclAllReduce'\r\n> \r\n> During handling of the above exception, another exception occurred:\r\n> \r\n> Traceback (most recent call last):\r\n>   File \"Z:/PyProj/leaf/othertest.py\", line 66, in <module>\r\n>     tf.app.run(argv=sys.argv)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\platform\\app.py\", line 125, in run\r\n>     _sys.exit(main(argv))\r\n>   File \"Z:/PyProj/leaf/othertest.py\", line 61, in main\r\n>     keras_estimator.train(input_fn=input_fn, steps=10)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\estimator\\estimator.py\", line 366, in train\r\n>     loss = self._train_model(input_fn, hooks, saving_listeners)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\estimator\\estimator.py\", line 1117, in _train_model\r\n>     return self._train_model_distributed(input_fn, hooks, saving_listeners)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\estimator\\estimator.py\", line 1160, in _train_model_distributed\r\n>     self.config)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\training\\distribute.py\", line 794, in call_for_each_tower\r\n>     return self._call_for_each_tower(fn, *args, **kwargs)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\distribute\\python\\mirrored_strategy.py\", line 269, in _call_for_each_tower\r\n>     coord.join(threads)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\training\\coordinator.py\", line 389, in join\r\n>     six.reraise(*self._exc_info_to_raise)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\six.py\", line 693, in reraise\r\n>     raise value\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\training\\coordinator.py\", line 297, in stop_on_exception\r\n>     yield\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\distribute\\python\\mirrored_strategy.py\", line 263, in _call_for_each_tower\r\n>     self, *merge_args, **merge_kwargs)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\training\\optimizer.py\", line 652, in _distributed_apply\r\n>     reduced_grads = distribution.batch_reduce(\"sum\", grads_and_vars)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\training\\distribute.py\", line 840, in batch_reduce\r\n>     return self._batch_reduce(method_string, value_destination_pairs)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\distribute\\python\\mirrored_strategy.py\", line 310, in _batch_reduce\r\n>     value_destination_pairs)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\distribute\\python\\cross_tower_ops.py\", line 177, in batch_reduce\r\n>     return self._batch_reduce(method_string, value_destination_pairs)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\distribute\\python\\cross_tower_ops.py\", line 475, in _batch_reduce\r\n>     [v[0] for v in value_destination_pairs])\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\distribute\\python\\cross_tower_ops.py\", line 520, in _batch_all_reduce\r\n>     device_grad_packs)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\distribute\\python\\cross_tower_utils.py\", line 37, in aggregate_gradients_using_nccl\r\n>     agg_grads = nccl.all_sum(single_grads)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\nccl\\python\\ops\\nccl_ops.py\", line 47, in all_sum\r\n>     return _apply_all_reduce('sum', tensors)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\nccl\\python\\ops\\nccl_ops.py\", line 228, in _apply_all_reduce\r\n>     shared_name=shared_name))\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\nccl\\ops\\gen_nccl_ops.py\", line 58, in nccl_all_reduce\r\n>     num_devices=num_devices, shared_name=shared_name, name=name)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py\", line 787, in _apply_op_helper\r\n>     op_def=op_def)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 3414, in create_op\r\n>     op_def=op_def)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 1756, in __init__\r\n>     control_input_ops)\r\n>   File \"D:\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 1592, in _create_c_op\r\n>     raise ValueError(str(e))\r\n> ValueError: Op type not registered 'NcclAllReduce' in binary running on DESKTO Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed. while building NodeDef 'NcclAllReduce'\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/417692473","html_url":"https://github.com/tensorflow/tensorflow/issues/21470#issuecomment-417692473","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21470","id":417692473,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNzY5MjQ3Mw==","user":{"login":"tensorflowbutler","id":28546240,"node_id":"MDQ6VXNlcjI4NTQ2MjQw","avatar_url":"https://avatars2.githubusercontent.com/u/28546240?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflowbutler","html_url":"https://github.com/tensorflowbutler","followers_url":"https://api.github.com/users/tensorflowbutler/followers","following_url":"https://api.github.com/users/tensorflowbutler/following{/other_user}","gists_url":"https://api.github.com/users/tensorflowbutler/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflowbutler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflowbutler/subscriptions","organizations_url":"https://api.github.com/users/tensorflowbutler/orgs","repos_url":"https://api.github.com/users/tensorflowbutler/repos","events_url":"https://api.github.com/users/tensorflowbutler/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflowbutler/received_events","type":"User","site_admin":false},"created_at":"2018-08-31T15:03:07Z","updated_at":"2018-08-31T15:03:07Z","author_association":"MEMBER","body":"Nagging Assignee @yuefengz: It has been 20 days with no activity and this issue has an assignee. Please update the label and/or status accordingly."}}
{"actor":{"display_login":"tensorflowbutler"}}