{"action":"opened","issue":{"url":"https://api.github.com/repos/Qihoo360/XLearning/issues/27","repository_url":"https://api.github.com/repos/Qihoo360/XLearning","labels_url":"https://api.github.com/repos/Qihoo360/XLearning/issues/27/labels{/name}","comments_url":"https://api.github.com/repos/Qihoo360/XLearning/issues/27/comments","events_url":"https://api.github.com/repos/Qihoo360/XLearning/issues/27/events","html_url":"https://github.com/Qihoo360/XLearning/issues/27","id":313595777,"number":27,"title":"在gpu-beta版本中提交到yarn申请资源超时","user":{"login":"zhanglistar","id":985418,"avatar_url":"https://avatars2.githubusercontent.com/u/985418?v=4","gravatar_id":"","url":"https://api.github.com/users/zhanglistar","html_url":"https://github.com/zhanglistar","followers_url":"https://api.github.com/users/zhanglistar/followers","following_url":"https://api.github.com/users/zhanglistar/following{/other_user}","gists_url":"https://api.github.com/users/zhanglistar/gists{/gist_id}","starred_url":"https://api.github.com/users/zhanglistar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhanglistar/subscriptions","organizations_url":"https://api.github.com/users/zhanglistar/orgs","repos_url":"https://api.github.com/users/zhanglistar/repos","events_url":"https://api.github.com/users/zhanglistar/events{/privacy}","received_events_url":"https://api.github.com/users/zhanglistar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-12T06:42:36Z","updated_at":"2018-04-12T06:42:36Z","closed_at":null,"author_association":"NONE","body":"集群是有资源的，\r\nAM log：\r\n`18/04/12 10:53:03 INFO ResourceUtils: Adding resource type - name = yarn.io/gpu, units = , type = COUNTABLE\r\n18/04/12 10:53:04 INFO Utilities: input path: hdfs://gpu1:8020/user/tmp/data/tensorflow\r\n18/04/12 10:53:04 INFO ApplicationMaster: XLearning application needs 2 worker and 1 ps  containers in fact\r\n18/04/12 10:53:04 INFO ApplicationMaster: Create worker container request: Capability[<memory:4096, vCores:2, yarn.io/gpu: 2>]Priority[3]AllocationRequestId[0]ExecutionTypeRequest[{Execution Type: GUARANTEED, Enforce Execution Type: false}]Resource Profile[null]\r\n18/04/12 10:53:04 INFO ApplicationMaster: Create ps container request: Capability[<memory:4096, vCores:2>]Priority[3]AllocationRequestId[0]ExecutionTypeRequest[{Execution Type: GUARANTEED, Enforce Execution Type: false}]Resource Profile[null]\r\n18/04/12 10:53:04 INFO ApplicationMaster: Try to allocate 1 ps/server containers\r\n18/04/12 10:53:05 INFO RMCallbackHandler: Acquired container container_1523451270416_0005_01_000002 on host gpu3 , with the resource <memory:4096, vCores:2>\r\n18/04/12 10:53:05 INFO RMCallbackHandler: Current acquired worker container 0 / 2 ps container 1 / 1\r\n18/04/12 10:53:06 INFO ApplicationMaster: Total 1 ps containers has allocated.\r\n18/04/12 10:53:06 INFO ApplicationMaster: Try to allocate 2 worker containers\r\n18/04/12 10:53:07 INFO RMCallbackHandler: Acquired container container_1523451270416_0005_01_000003 on host gpu3 , with the resource <memory:4096, vCores:2, yarn.io/gpu: 2>\r\n18/04/12 10:53:07 INFO RMCallbackHandler: Current acquired worker container 1 / 2 ps container 1 / 1\r\n18/04/12 11:03:08 INFO ApplicationMaster: Container waiting except the allocated expiry time. Maybe the Cluster available resources are not satisfied the user need. Please resubmit !\r\n18/04/12 11:03:08 INFO ApplicationMaster: Unregister  Application\r\n18/04/12 11:03:08 INFO AMRMClientImpl: Waiting for application to be successfully unregistered.\r\n18/04/12 11:03:08 INFO ApplicationMaster: Application failed.\r\n`\r\nResourceManager log：\r\n`clusterResource=<memory:400000, vCores:36, yarn.io/gpu: 8> `\r\n\r\nrun.sh:\r\n` --worker-memory 4G \\\r\n   --worker-num 2 \\\r\n   --worker-cores 2 \\\r\n   --worker-gcores 2 \\\r\n   --ps-memory 4G \\\r\n   --ps-num 1 \\\r\n   --ps-cores 2 \\\r\n`"}}
{"actor":{"display_login":"zhanglistar"}}