{"action":"closed","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2456","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2456/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2456/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2456/events","html_url":"https://github.com/tensorflow/tensorflow/issues/2456","id":156112744,"number":2456,"title":"Problem in distributed parameter server","user":{"login":"perhapszzy","id":7953637,"avatar_url":"https://avatars.githubusercontent.com/u/7953637?v=3","gravatar_id":"","url":"https://api.github.com/users/perhapszzy","html_url":"https://github.com/perhapszzy","followers_url":"https://api.github.com/users/perhapszzy/followers","following_url":"https://api.github.com/users/perhapszzy/following{/other_user}","gists_url":"https://api.github.com/users/perhapszzy/gists{/gist_id}","starred_url":"https://api.github.com/users/perhapszzy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/perhapszzy/subscriptions","organizations_url":"https://api.github.com/users/perhapszzy/orgs","repos_url":"https://api.github.com/users/perhapszzy/repos","events_url":"https://api.github.com/users/perhapszzy/events{/privacy}","received_events_url":"https://api.github.com/users/perhapszzy/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-05-21T17:26:48Z","updated_at":"2016-05-23T18:50:27Z","closed_at":"2016-05-23T18:50:27Z","body":"### Environment info\r\nRunning the distributed tensorflow on GCE kubernetes cluster using the [demo](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/dist_test/scripts/dist_mnist_test.sh) with the default images.\r\n\r\n### Steps to reproduce\r\n1. Run the demo with [hidden_units](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/dist_test/python/mnist_replica.py#L69) = 100\r\n2. Run the demo with  [hidden_units](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/dist_test/python/mnist_replica.py#L69) = 500\r\n\r\n### What have you tried?\r\n1. Restart all the pods (this worked)\r\n\r\n### Logs or other output that would be helpful\r\n```\r\nTraceback (most recent call last):\r\n  File \"/home/zeyu/gocode/src/github.com/caicloud/BigData/distributed-tensorflow/runner/demo/mnist_replica.py\", line 250, in <module>\r\n    tf.app.run()\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py\", line 30, in run\r\n    sys.exit(main(sys.argv))\r\n  File \"/home/zeyu/gocode/src/github.com/caicloud/BigData/distributed-tensorflow/runner/demo/mnist_replica.py\", line 203, in main\r\n    with sv.prepare_or_wait_for_session(FLAGS.worker_grpc_url, config=sess_config) as sess:\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py\", line 685, in prepare_or_wait_for_session\r\n    config=config, init_feed_dict=self._init_feed_dict)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py\", line 163, in prepare_session\r\n    sess.run(init_op, feed_dict=init_feed_dict)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 340, in run\r\n    run_metadata_ptr)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 564, in _run\r\n    feed_dict_string, options, run_metadata)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 637, in _do_run\r\n    target_list, options, run_metadata)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 659, in _do_call\r\n    e.code)\r\ntensorflow.python.framework.errors.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [100,10] rhs shape= [500,10]\r\n\t [[Node: layer2/weights/Variable/Assign = Assign[T=DT_FLOAT, _class=[\"loc:@layer2/weights/Variable\"], use_locking=true, validate_shape=true, _device=\"/job:ps/replica:0/task:1/cpu:0\"](layer2/weights/Variable, layer2/weights/truncated_normal_S19)]]\r\nCaused by op u'layer2/weights/Variable/Assign', defined at:\r\n  File \"/home/zeyu/gocode/src/github.com/caicloud/BigData/distributed-tensorflow/runner/demo/mnist_replica.py\", line 250, in <module>\r\n    tf.app.run()\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py\", line 30, in run\r\n    sys.exit(main(sys.argv))\r\n  File \"/home/zeyu/gocode/src/github.com/caicloud/BigData/distributed-tensorflow/runner/demo/mnist_replica.py\", line 157, in main\r\n    y = nn_layer(hidden1, FLAGS.hidden_units, 10, 'layer2', act=tf.nn.softmax)\r\n  File \"/home/zeyu/gocode/src/github.com/caicloud/BigData/distributed-tensorflow/runner/demo/mnist_replica.py\", line 115, in nn_layer\r\n    weights = weight_variable([input_dim, output_dim])\r\n  File \"/home/zeyu/gocode/src/github.com/caicloud/BigData/distributed-tensorflow/runner/demo/mnist_replica.py\", line 85, in weight_variable\r\n    return tf.Variable(initial)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py\", line 209, in __init__\r\n    dtype=dtype)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py\", line 308, in _init_from_args\r\n    validate_shape=validate_shape).op\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_state_ops.py\", line 40, in assign\r\n    use_locking=use_locking, name=name)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py\", line 655, in apply_op\r\n    op_def=op_def)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py\", line 2154, in create_op\r\n    original_op=self._default_original_op, op_def=op_def)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py\", line 1154, in __init__\r\n    self._traceback = _extract_stack()\r\n```\r\n\r\nThe error happened at the init time, and it seems that the log suggests the parameter server didn't clean up the information from previous job:\r\n\r\n```\r\ntensorflow.python.framework.errors.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [100,10] rhs shape= [500,10]\r\n```\r\n\r\n"}}
{"actor":{"display_login":"vrv"}}