{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensor2tensor/issues/266","repository_url":"https://api.github.com/repos/tensorflow/tensor2tensor","labels_url":"https://api.github.com/repos/tensorflow/tensor2tensor/issues/266/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensor2tensor/issues/266/comments","events_url":"https://api.github.com/repos/tensorflow/tensor2tensor/issues/266/events","html_url":"https://github.com/tensorflow/tensor2tensor/issues/266","id":254168774,"number":266,"title":"Evaluation failed when training with multiple work gpus","user":{"login":"skyw","id":1697840,"avatar_url":"https://avatars3.githubusercontent.com/u/1697840?v=4","gravatar_id":"","url":"https://api.github.com/users/skyw","html_url":"https://github.com/skyw","followers_url":"https://api.github.com/users/skyw/followers","following_url":"https://api.github.com/users/skyw/following{/other_user}","gists_url":"https://api.github.com/users/skyw/gists{/gist_id}","starred_url":"https://api.github.com/users/skyw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/skyw/subscriptions","organizations_url":"https://api.github.com/users/skyw/orgs","repos_url":"https://api.github.com/users/skyw/repos","events_url":"https://api.github.com/users/skyw/events{/privacy}","received_events_url":"https://api.github.com/users/skyw/received_events","type":"User","site_admin":false},"labels":[{"id":627478123,"url":"https://api.github.com/repos/tensorflow/tensor2tensor/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":35,"created_at":"2017-08-30T23:53:29Z","updated_at":"2018-02-02T23:10:53Z","closed_at":"2017-12-22T16:21:27Z","author_association":"NONE","body":"I set worker_gpu to 2 and use 2 gpus in the same node. training is completely fine. But evaluation fails with this error\r\n\"\r\ntensorflow.python.framework.errors_impl.InvalidArgumentError: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 47) and num_split 2\r\n         [[Node: split_2 = Split[T=DT_INT32, num_split=2, _device=\"/job:localhost/replica:0/task:0/cpu:0\"](split_2/split_dim, input_reader/ExpandDims_3/_1823)]]\r\n         [[Node: split_2/_1825 = _HostRecv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/gpu:1\", send_device=\"/job:localhost/replica:0/task:0/cpu:0\", send_device_incarnation=1, tensor_name=\"edge_1113_split_2\", tensor_type=DT_INT32, _device=\"/job:localhost/replica:0/task:0/gpu:1\"]()]]\r\n\r\nCaused by op u'split_2', defined at:\r\n  File \"/raid/skyw/venv/tensorflow-pip-py27/bin/t2t-trainer\", line 5, in <module>\r\n    pkg_resources.run_script('tensor2tensor==1.2.1', 't2t-trainer')\r\n\"\r\n\r\nIt wasn't very clear what the error is. But single GPU training/evaluation is fine, the problem comes with 2 worker GPUs."},"comment":{"url":"https://api.github.com/repos/tensorflow/tensor2tensor/issues/comments/362736065","html_url":"https://github.com/tensorflow/tensor2tensor/issues/266#issuecomment-362736065","issue_url":"https://api.github.com/repos/tensorflow/tensor2tensor/issues/266","id":362736065,"user":{"login":"martinpopel","id":724617,"avatar_url":"https://avatars1.githubusercontent.com/u/724617?v=4","gravatar_id":"","url":"https://api.github.com/users/martinpopel","html_url":"https://github.com/martinpopel","followers_url":"https://api.github.com/users/martinpopel/followers","following_url":"https://api.github.com/users/martinpopel/following{/other_user}","gists_url":"https://api.github.com/users/martinpopel/gists{/gist_id}","starred_url":"https://api.github.com/users/martinpopel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinpopel/subscriptions","organizations_url":"https://api.github.com/users/martinpopel/orgs","repos_url":"https://api.github.com/users/martinpopel/repos","events_url":"https://api.github.com/users/martinpopel/events{/privacy}","received_events_url":"https://api.github.com/users/martinpopel/received_events","type":"User","site_admin":false},"created_at":"2018-02-02T23:10:53Z","updated_at":"2018-02-02T23:10:53Z","author_association":"CONTRIBUTOR","body":"For me the internal evaluation works out of the box on 8 GPUs in T2T v1.4.2. There are 8 times more warnings \"*W tensorflow/core/framework/op_kernel.cc:1192] Out of range: End of sequence*\", but these are just warnings (and OK in this case), not errors. I can see approx_bleu, rouge etc. in Tensorboard.\r\nSo technically the internal evaluation works, although there are some related issues:\r\n* As discussed in #444, scaling optimal hyperparameters to more GPUs (and thus higher effective batch size) is not \"out of the box\".\r\n* As discussed in #495, with the default `schedule=continuous_train_and_eval` two checkpoints are saved instead of one (e.g. 2000 and 2001).\r\n* As discussed in #407, approx_bleu used to be wrong/not reliable. I am not sure if this has been improved (I guess no), I am using `t2t-bleu` instead. "}}
{"actor":{"display_login":"martinpopel"}}