{"action":"opened","issue":{"url":"https://api.github.com/repos/tensorflow/models/issues/56","repository_url":"https://api.github.com/repos/tensorflow/models","labels_url":"https://api.github.com/repos/tensorflow/models/issues/56/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/models/issues/56/comments","events_url":"https://api.github.com/repos/tensorflow/models/issues/56/events","html_url":"https://github.com/tensorflow/models/issues/56","id":151313512,"number":56,"title":"Error running multiple workers on one machine which has multiple GPUs","user":{"login":"ZhuFengdaaa","id":9649227,"avatar_url":"https://avatars.githubusercontent.com/u/9649227?v=3","gravatar_id":"","url":"https://api.github.com/users/ZhuFengdaaa","html_url":"https://github.com/ZhuFengdaaa","followers_url":"https://api.github.com/users/ZhuFengdaaa/followers","following_url":"https://api.github.com/users/ZhuFengdaaa/following{/other_user}","gists_url":"https://api.github.com/users/ZhuFengdaaa/gists{/gist_id}","starred_url":"https://api.github.com/users/ZhuFengdaaa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ZhuFengdaaa/subscriptions","organizations_url":"https://api.github.com/users/ZhuFengdaaa/orgs","repos_url":"https://api.github.com/users/ZhuFengdaaa/repos","events_url":"https://api.github.com/users/ZhuFengdaaa/events{/privacy}","received_events_url":"https://api.github.com/users/ZhuFengdaaa/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-04-27T07:03:20Z","updated_at":"2016-04-27T07:03:20Z","closed_at":null,"body":"I run the distribute version of the [example](https://github.com/tensorflow/models/tree/master/inception)\r\non two machine, each of them has 4 GPUs. if I start one worker each, its OK. However, when I want to run multiple workers on each machine, **\"InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [4] rhs shape= [2]\"** exist. \r\n\r\n```\r\n# machine A with 4 GPU\r\n~/models/inception/bazel-bin/inception/imagenet_distributed_train \\\r\n--batch_size=32 \\\r\n--data_dir=/data1/imagenet1k \\\r\n--job_name='worker' \\\r\n--task_id=0 \\\r\n--ps_hosts='10.10.102.28:2220' \\\r\n--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.29:2221,10.10.102.29:2222'\r\n\r\n~/models/inception/bazel-bin/inception/imagenet_distributed_train \\\r\n--batch_size=32 \\\r\n--data_dir=/data1/imagenet1k \\\r\n--job_name='worker' \\\r\n--task_id=1 \\\r\n--ps_hosts='10.10.102.28:2220' \\\r\n--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.29:2221,10.10.102.29:2222'\r\n\r\n~/models/inception/bazel-bin/inception/imagenet_distributed_train \\\r\n--job_name='ps' \\\r\n-task_id=0 \\\r\n--ps_hosts='10.10.102.28:2220' \\\r\n--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.29:2221,10.10.102.29:2222'\r\n\r\n# machine B with 4 GPU\r\n~/models/inception/bazel-bin/inception/imagenet_distributed_train \\\r\n--batch_size=32 \\\r\n--data_dir=/data1/imagenet1k \\\r\n--job_name='worker' \\\r\n--task_id=2 \\\r\n--ps_hosts='10.10.102.28:2220' \\\r\n--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.29:2221,10.10.102.29:2222'\r\n\r\n~/models/inception/bazel-bin/inception/imagenet_distributed_train \\\r\n--batch_size=32 \\\r\n--data_dir=/data1/imagenet1k \\\r\n--job_name='worker' \\\r\n--task_id=3 \\\r\n--ps_hosts='10.10.102.28:2220' \\\r\n--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.29:2221,10.10.102.29:2222'\r\n```\r\n\r\n**Error log**\r\n\r\n```\r\nINFO:tensorflow:Waiting for model to be ready: Attempting to use uninitialized value mixed_35x35x256a/branch3x3dbl/Conv/weights/ExponentialMovingAverage\r\n\t [[Node: mixed_35x35x256a/branch3x3dbl/Conv/weights/ExponentialMovingAverage/read = Identity[T=DT_FLOAT, _class=[\"loc:@mixed_35x35x256a/branch3x3dbl/Conv/weights\"], _device=\"/job:ps/replica:0/task:0/cpu:0\"](mixed_35x35x256a/branch3x3dbl/Conv/weights/ExponentialMovingAverage)]]\r\nCaused by op u'mixed_35x35x256a/branch3x3dbl/Conv/weights/ExponentialMovingAverage/read', defined at:\r\n  File \"/home/models/inception/bazel-bin/inception/imagenet_distributed_train.runfiles/__main__/inception/imagenet_distributed_train.py\", line 65, in <module>\r\n    tf.app.run()\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py\", line 30, in run\r\n    sys.exit(main(sys.argv))\r\n  File \"/home/models/inception/bazel-bin/inception/imagenet_distributed_train.runfiles/__main__/inception/imagenet_distributed_train.py\", line 61, in main\r\n    inception_distributed_train.train(server.target, dataset, cluster_spec)\r\n  File \"/home/models/inception/bazel-bin/inception/imagenet_distributed_train.runfiles/__main__/inception/inception_distributed_train.py\", line 220, in train\r\n    apply_gradients_op = opt.apply_gradients(grads, global_step=global_step)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/training/sync_replicas_optimizer.py\", line 427, in apply_gradients\r\n    self._variables_to_average)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/training/moving_averages.py\", line 282, in apply\r\n    colocate_with_primary=True)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/training/slot_creator.py\", line 86, in create_slot\r\n    return _create_slot_var(primary, val, scope)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/training/slot_creator.py\", line 50, in _create_slot_var\r\n    slot = variables.Variable(val, name=scope, trainable=False)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/ops/variables.py\", line 206, in __init__\r\n    dtype=dtype)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/ops/variables.py\", line 275, in _init_from_args\r\n    self._snapshot = array_ops.identity(self._variable, name=\"read\")\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py\", line 609, in identity\r\n    return _op_def_lib.apply_op(\"Identity\", input=input, name=name)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/ops/op_def_library.py\", line 655, in apply_op\r\n    op_def=op_def)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py\", line 2154, in create_op\r\n    original_op=self._default_original_op, op_def=op_def)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py\", line 1154, in __init__\r\n    self._traceback = _extract_stack()\r\n\r\nTraceback (most recent call last):\r\n  File \"/home/models/inception/bazel-bin/inception/imagenet_distributed_train.runfiles/__main__/inception/imagenet_distributed_train.py\", line 65, in <module>\r\n    tf.app.run()\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py\", line 30, in run\r\n    sys.exit(main(sys.argv))\r\n  File \"/home/models/inception/bazel-bin/inception/imagenet_distributed_train.runfiles/__main__/inception/imagenet_distributed_train.py\", line 61, in main\r\n    inception_distributed_train.train(server.target, dataset, cluster_spec)\r\n  File \"/home/models/inception/bazel-bin/inception/imagenet_distributed_train.runfiles/__main__/inception/inception_distributed_train.py\", line 260, in train\r\n    sess = sv.prepare_or_wait_for_session(target, config=sess_config)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py\", line 674, in prepare_or_wait_for_session\r\n    config=config, init_feed_dict=self._init_feed_dict)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py\", line 158, in prepare_session\r\n    max_wait_secs=max_wait_secs, config=config)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py\", line 214, in recover_session\r\n    saver.restore(sess, ckpt.model_checkpoint_path)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py\", line 1090, in restore\r\n    {self.saver_def.filename_tensor_name: save_path})\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 340, in run\r\n    run_metadata_ptr)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 564, in _run\r\n    feed_dict_string, options, run_metadata)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 637, in _do_run\r\n    target_list, options, run_metadata)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 659, in _do_call\r\n    e.code)\r\ntensorflow.python.framework.errors.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [4] rhs shape= [2]\r\n\t [[Node: save/Assign_103 = Assign[T=DT_INT64, _class=[\"loc:@local_steps\"], use_locking=true, validate_shape=true, _device=\"/job:ps/replica:0/task:0/cpu:0\"](local_steps, save/restore_slice_103)]]\r\n\t [[Node: save/restore_all/NoOp_S4 = _Recv[client_terminated=false, recv_device=\"/job:worker/replica:0/task:0/gpu:0\", send_device=\"/job:ps/replica:0/task:0/cpu:0\", send_device_incarnation=1831303354831316628, tensor_name=\"edge_1174_save/restore_all/NoOp\", tensor_type=DT_FLOAT, _device=\"/job:worker/replica:0/task:0/gpu:0\"]()]]\r\nCaused by op u'save/Assign_103', defined at:\r\n  File \"/home/models/inception/bazel-bin/inception/imagenet_distributed_train.runfiles/__main__/inception/imagenet_distributed_train.py\", line 65, in <module>\r\n    tf.app.run()\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py\", line 30, in run\r\n    sys.exit(main(sys.argv))\r\n  File \"/home/models/inception/bazel-bin/inception/imagenet_distributed_train.runfiles/__main__/inception/imagenet_distributed_train.py\", line 61, in main\r\n    inception_distributed_train.train(server.target, dataset, cluster_spec)\r\n  File \"/home/models/inception/bazel-bin/inception/imagenet_distributed_train.runfiles/__main__/inception/inception_distributed_train.py\", line 233, in train\r\n    saver = tf.train.Saver()\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py\", line 832, in __init__\r\n    restore_sequentially=restore_sequentially)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py\", line 502, in build\r\n    filename_tensor, vars_to_save, restore_sequentially, reshape)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py\", line 268, in _AddRestoreOps\r\n    validate_shape=validate_shape))\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_state_ops.py\", line 40, in assign\r\n    use_locking=use_locking, name=name)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/ops/op_def_library.py\", line 655, in apply_op\r\n    op_def=op_def)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py\", line 2154, in create_op\r\n    original_op=self._default_original_op, op_def=op_def)\r\n  File \"/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py\", line 1154, in __init__\r\n    self._traceback = _extract_stack()\r\n\r\n```\r\n"}}