{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/23376","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/23376/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/23376/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/23376/events","html_url":"https://github.com/tensorflow/tensorflow/issues/23376","id":375491400,"node_id":"MDU6SXNzdWUzNzU0OTE0MDA=","number":23376,"title":"ParameterServerStrategy throws an exception when training locally with multiple GPUS","user":{"login":"eryshev","id":7483130,"node_id":"MDQ6VXNlcjc0ODMxMzA=","avatar_url":"https://avatars3.githubusercontent.com/u/7483130?v=4","gravatar_id":"","url":"https://api.github.com/users/eryshev","html_url":"https://github.com/eryshev","followers_url":"https://api.github.com/users/eryshev/followers","following_url":"https://api.github.com/users/eryshev/following{/other_user}","gists_url":"https://api.github.com/users/eryshev/gists{/gist_id}","starred_url":"https://api.github.com/users/eryshev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eryshev/subscriptions","organizations_url":"https://api.github.com/users/eryshev/orgs","repos_url":"https://api.github.com/users/eryshev/repos","events_url":"https://api.github.com/users/eryshev/events{/privacy}","received_events_url":"https://api.github.com/users/eryshev/received_events","type":"User","site_admin":false},"labels":[{"id":1097547538,"node_id":"MDU6TGFiZWwxMDk3NTQ3NTM4","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/comp:gpu","name":"comp:gpu","color":"0052cc","default":false},{"id":404586594,"node_id":"MDU6TGFiZWw0MDQ1ODY1OTQ=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"f4b400","default":false},{"id":473172988,"node_id":"MDU6TGFiZWw0NzMxNzI5ODg=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:bug/performance","name":"type:bug/performance","color":"159b2e","default":false}],"state":"open","locked":false,"assignee":{"login":"yuefengz","id":1647833,"node_id":"MDQ6VXNlcjE2NDc4MzM=","avatar_url":"https://avatars0.githubusercontent.com/u/1647833?v=4","gravatar_id":"","url":"https://api.github.com/users/yuefengz","html_url":"https://github.com/yuefengz","followers_url":"https://api.github.com/users/yuefengz/followers","following_url":"https://api.github.com/users/yuefengz/following{/other_user}","gists_url":"https://api.github.com/users/yuefengz/gists{/gist_id}","starred_url":"https://api.github.com/users/yuefengz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yuefengz/subscriptions","organizations_url":"https://api.github.com/users/yuefengz/orgs","repos_url":"https://api.github.com/users/yuefengz/repos","events_url":"https://api.github.com/users/yuefengz/events{/privacy}","received_events_url":"https://api.github.com/users/yuefengz/received_events","type":"User","site_admin":false},"assignees":[{"login":"yuefengz","id":1647833,"node_id":"MDQ6VXNlcjE2NDc4MzM=","avatar_url":"https://avatars0.githubusercontent.com/u/1647833?v=4","gravatar_id":"","url":"https://api.github.com/users/yuefengz","html_url":"https://github.com/yuefengz","followers_url":"https://api.github.com/users/yuefengz/followers","following_url":"https://api.github.com/users/yuefengz/following{/other_user}","gists_url":"https://api.github.com/users/yuefengz/gists{/gist_id}","starred_url":"https://api.github.com/users/yuefengz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yuefengz/subscriptions","organizations_url":"https://api.github.com/users/yuefengz/orgs","repos_url":"https://api.github.com/users/yuefengz/repos","events_url":"https://api.github.com/users/yuefengz/events{/privacy}","received_events_url":"https://api.github.com/users/yuefengz/received_events","type":"User","site_admin":false},{"login":"azaks2","id":40365382,"node_id":"MDQ6VXNlcjQwMzY1Mzgy","avatar_url":"https://avatars2.githubusercontent.com/u/40365382?v=4","gravatar_id":"","url":"https://api.github.com/users/azaks2","html_url":"https://github.com/azaks2","followers_url":"https://api.github.com/users/azaks2/followers","following_url":"https://api.github.com/users/azaks2/following{/other_user}","gists_url":"https://api.github.com/users/azaks2/gists{/gist_id}","starred_url":"https://api.github.com/users/azaks2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/azaks2/subscriptions","organizations_url":"https://api.github.com/users/azaks2/orgs","repos_url":"https://api.github.com/users/azaks2/repos","events_url":"https://api.github.com/users/azaks2/events{/privacy}","received_events_url":"https://api.github.com/users/azaks2/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2018-10-30T13:34:17Z","updated_at":"2018-11-05T08:43:35Z","closed_at":null,"author_association":"NONE","body":"**System information**\r\n- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No\r\n- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): centos-release-7-4.1708.el7.centos.x86_64\r\n- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No\r\n- TensorFlow installed from (source or binary): compiled form source with flags \r\n```\r\nexport TF_NEED_HDFS=1\r\nexport TF_NEED_KAFKA=1\r\nexport TF_ENABLE_XLA=1\r\nexport TF_NEED_CUDA=1\r\nexport TF_CUDA_COMPUTE_CAPABILITIES=5.2,7.0\r\nexport TF_CUDA_VERSION=9.2\r\nexport TF_CUDNN_VERSION=7\r\nexport TF_NCCL_VERSION=1.3\r\n\r\nbazel build \\\r\n            --config=opt \\\r\n            --config=cuda \\\r\n            --copt=-msse4.2 \\\r\n            --copt=-mavx \\\r\n            --copt=-mavx2 \\\r\n            --copt=-mfma \\\r\n            --copt=-O3 \\\r\n            //tensorflow/tools/pip_package:build_pip_package\r\n```\r\n- TensorFlow version (use command below): b'v1.11.0-0-gc19e293' 1.11.0\r\n- Python version: Python 3.6.6\r\n- Bazel version (if compiling from source): 0.18.0\r\n- GCC/Compiler version (if compiling from source): 4.8.5 20150623\r\n- CUDA/cuDNN version: 9.2, 7\r\n- GPU model and memory: Tesla V100-PCIE 16GB and Tesla V100-PCIE 16GB\r\n\r\n**Describe the current behavior**\r\nI'm trying to run an official resnet model on cifar 10 dataset from https://github.com/tensorflow/models/tree/master/official/resnet \r\nwhere I replaced MirroredStrategy by ParameterServerStrategy in https://github.com/tensorflow/models/blob/master/official/utils/misc/distribution_utils.py#L48\r\nas following\r\n```\r\n# return tf.contrib.distribute.MirroredStrategy(num_gpus=num_gpus)\r\n     return tf.contrib.distribute.ParameterServerStrategy(num_gpus)\r\n```\r\n\r\nIt throws an exception\r\n```\r\nI1030 10:21:32.401904 139688013772544 tf_logging.py:115] Done calling model_fn.\r\nTraceback (most recent call last):\r\n  File \"./models/official/resnet/cifar10_main.py\", line 285, in <module>\r\n    absl_app.run(main)\r\n  File \"/home/a.eryshev/miniconda3/envs/py36/lib/python3.6/site-packages/absl/app.py\", line 300, in run\r\n    _run_main(main, args)\r\n  File \"/home/a.eryshev/miniconda3/envs/py36/lib/python3.6/site-packages/absl/app.py\", line 251, in _run_main\r\n    sys.exit(main(argv))\r\n  File \"./models/official/resnet/cifar10_main.py\", line 278, in main\r\n    run_cifar(flags.FLAGS)\r\n  File \"./models/official/resnet/cifar10_main.py\", line 273, in run_cifar\r\n    shape=[_HEIGHT, _WIDTH, _NUM_CHANNELS])\r\n  File \"/home/a.eryshev/dev/tf-experiments/models/official/resnet/resnet_run_loop.py\", line 565, in resnet_main\r\n    hooks=train_hooks, max_steps=flags_obj.max_train_steps)\r\n  File \"/home/a.eryshev/miniconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 356, in train\r\n    loss = self._train_model(input_fn, hooks, saving_listeners)\r\n  File \"/home/a.eryshev/miniconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 1179, in _train_model\r\n    return self._train_model_distributed(input_fn, hooks, saving_listeners)\r\n  File \"/home/a.eryshev/miniconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 1295, in _train_model_distributed\r\n    destinations='/device:CPU:0'))[0]\r\n  File \"/home/a.eryshev/miniconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/training/distribute.py\", line 748, in reduce\r\n    return self._reduce(aggregation, value, destinations)\r\n  File \"/home/a.eryshev/miniconda3/envs/py36/lib/python3.6/site-packages/tensorflow/contrib/distribute/python/parameter_server_strategy.py\", line 305, in _reduce\r\n    self._verify_destinations_not_different_worker(destinations)\r\n  File \"/home/a.eryshev/miniconda3/envs/py36/lib/python3.6/site-packages/tensorflow/contrib/distribute/python/parameter_server_strategy.py\", line 302, in _verify_destinations_not_different_worker\r\n    (d, self._worker_device))\r\nAttributeError: 'ParameterServerStrategy' object has no attribute '_worker_device'\r\n```\r\n\r\n**Describe the expected behavior**\r\nReplacing MirroredStrategy by ParameterServerStrategy launches a training with CPU acting as PS and 2 GPU workers.\r\n\r\n**Code to reproduce the issue**\r\nRun command:\r\n```\r\npython ./models/official/resnet/cifar10_main.py --data_dir=/data --model_dir=/data --benchmark_logger_type=BaseBenchmarkLogger --resnet_version=2 --clean --log_step_count_steps=10 --save_summary_steps=10 --epochs_between_evals=1820 --train_epochs=1820 --batch_size=512 --num_gpus=2\r\n```"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/435795396","html_url":"https://github.com/tensorflow/tensorflow/issues/23376#issuecomment-435795396","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/23376","id":435795396,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNTc5NTM5Ng==","user":{"login":"eryshev","id":7483130,"node_id":"MDQ6VXNlcjc0ODMxMzA=","avatar_url":"https://avatars3.githubusercontent.com/u/7483130?v=4","gravatar_id":"","url":"https://api.github.com/users/eryshev","html_url":"https://github.com/eryshev","followers_url":"https://api.github.com/users/eryshev/followers","following_url":"https://api.github.com/users/eryshev/following{/other_user}","gists_url":"https://api.github.com/users/eryshev/gists{/gist_id}","starred_url":"https://api.github.com/users/eryshev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eryshev/subscriptions","organizations_url":"https://api.github.com/users/eryshev/orgs","repos_url":"https://api.github.com/users/eryshev/repos","events_url":"https://api.github.com/users/eryshev/events{/privacy}","received_events_url":"https://api.github.com/users/eryshev/received_events","type":"User","site_admin":false},"created_at":"2018-11-05T08:43:35Z","updated_at":"2018-11-05T08:43:35Z","author_association":"NONE","body":"Looks like there's an open PR that corrects the problem https://github.com/tensorflow/tensorflow/pull/22713"}}
{"actor":{"display_login":"eryshev"}}