{"action":"closed","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3636","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3636/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3636/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3636/events","html_url":"https://github.com/tensorflow/tensorflow/issues/3636","id":169296650,"node_id":"MDU6SXNzdWUxNjkyOTY2NTA=","number":3636,"title":"Inception retraining / transfer learning can only utilize one CPU core and one GPU","user":{"login":"HadronCloud","id":13091475,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzMDkxNDc1","avatar_url":"https://avatars0.githubusercontent.com/u/13091475?v=4","gravatar_id":"","url":"https://api.github.com/users/HadronCloud","html_url":"https://github.com/HadronCloud","followers_url":"https://api.github.com/users/HadronCloud/followers","following_url":"https://api.github.com/users/HadronCloud/following{/other_user}","gists_url":"https://api.github.com/users/HadronCloud/gists{/gist_id}","starred_url":"https://api.github.com/users/HadronCloud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HadronCloud/subscriptions","organizations_url":"https://api.github.com/users/HadronCloud/orgs","repos_url":"https://api.github.com/users/HadronCloud/repos","events_url":"https://api.github.com/users/HadronCloud/events{/privacy}","received_events_url":"https://api.github.com/users/HadronCloud/received_events","type":"Organization","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"tfboyd","id":23486130,"node_id":"MDQ6VXNlcjIzNDg2MTMw","avatar_url":"https://avatars1.githubusercontent.com/u/23486130?v=4","gravatar_id":"","url":"https://api.github.com/users/tfboyd","html_url":"https://github.com/tfboyd","followers_url":"https://api.github.com/users/tfboyd/followers","following_url":"https://api.github.com/users/tfboyd/following{/other_user}","gists_url":"https://api.github.com/users/tfboyd/gists{/gist_id}","starred_url":"https://api.github.com/users/tfboyd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tfboyd/subscriptions","organizations_url":"https://api.github.com/users/tfboyd/orgs","repos_url":"https://api.github.com/users/tfboyd/repos","events_url":"https://api.github.com/users/tfboyd/events{/privacy}","received_events_url":"https://api.github.com/users/tfboyd/received_events","type":"User","site_admin":false},"assignees":[{"login":"tfboyd","id":23486130,"node_id":"MDQ6VXNlcjIzNDg2MTMw","avatar_url":"https://avatars1.githubusercontent.com/u/23486130?v=4","gravatar_id":"","url":"https://api.github.com/users/tfboyd","html_url":"https://github.com/tfboyd","followers_url":"https://api.github.com/users/tfboyd/followers","following_url":"https://api.github.com/users/tfboyd/following{/other_user}","gists_url":"https://api.github.com/users/tfboyd/gists{/gist_id}","starred_url":"https://api.github.com/users/tfboyd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tfboyd/subscriptions","organizations_url":"https://api.github.com/users/tfboyd/orgs","repos_url":"https://api.github.com/users/tfboyd/repos","events_url":"https://api.github.com/users/tfboyd/events{/privacy}","received_events_url":"https://api.github.com/users/tfboyd/received_events","type":"User","site_admin":false}],"milestone":null,"comments":13,"created_at":"2016-08-04T05:52:31Z","updated_at":"2018-10-12T21:48:01Z","closed_at":"2018-10-12T21:48:01Z","author_association":"NONE","body":"The retraining code is example code, but it does seem like there are many people experimenting with retraining and improving performance could make the example code much more useful as a launchpad for others if it can scale across GPUs and multiple CPU cores.\n\nAt the moment, running\n`bazel-bin/tensorflow/examples/image_retraining/retrain --num_gpus=2 --image_dir /images`\nwill only result in a single GPU performing computations in `nvidia-smi`, even though both GPUs are running `python` in nvidia-smi.  GPU #2 will see zero memory/power utilization during the computation beyond idle levels.  GPU #1 will utilize all memory TensorFlow and an additional 30 watts or so.\n\nThe other related issue is that, during training, a single CPU core will be maxed out at 100% running python; given the capabilities of the card, I'm guessing removing this bottleneck would increase retraining speed by perhaps 200-400%.  `nmon` shows that disk IO is not a bottleneck.\n\nThanks for your time and for making TensorFlow available to everyone!\n### Environment info\n\nOperating System: Ubuntu 14.04\n\nInstalled version of CUDA and cuDNN: 8.0 and 5 with two GTX 1080 cards\n\nretrain.py: `train_batch_size` is 20000 and `learning_rate` is 0.5\n1. The commit hash (`git rev-parse HEAD`): r0.9.0\n2. The output of `bazel version`: Build label: 0.2.3\n   Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar\n   Build time: Tue May 17 14:21:13 2016 (1463494873)\n   Build timestamp: 1463494873\n   Build timestamp as int: 1463494873\n"}}
{"actor":{"display_login":"dksb"}}