{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/1284","repository_url":"https://api.github.com/repos/dmlc/xgboost","labels_url":"https://api.github.com/repos/dmlc/xgboost/issues/1284/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/xgboost/issues/1284/comments","events_url":"https://api.github.com/repos/dmlc/xgboost/issues/1284/events","html_url":"https://github.com/dmlc/xgboost/issues/1284","id":161057144,"number":1284,"title":"[jvm-packages] Executors fail during prediction (Doing rabit call after Finalize)","user":{"login":"titicaca","id":1780746,"avatar_url":"https://avatars.githubusercontent.com/u/1780746?v=3","gravatar_id":"","url":"https://api.github.com/users/titicaca","html_url":"https://github.com/titicaca","followers_url":"https://api.github.com/users/titicaca/followers","following_url":"https://api.github.com/users/titicaca/following{/other_user}","gists_url":"https://api.github.com/users/titicaca/gists{/gist_id}","starred_url":"https://api.github.com/users/titicaca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/titicaca/subscriptions","organizations_url":"https://api.github.com/users/titicaca/orgs","repos_url":"https://api.github.com/users/titicaca/repos","events_url":"https://api.github.com/users/titicaca/events{/privacy}","received_events_url":"https://api.github.com/users/titicaca/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":22,"created_at":"2016-06-19T05:17:30Z","updated_at":"2016-07-05T13:06:10Z","closed_at":null,"body":"I was trying to predict data using the trained model in a spark cluster of 3 nodes with yarn, but executors failed by accident.  New executors will be started to continue the spark task. The spark task can be finished sucessfully and I can still get the correst predictions. However, after several times, the spark context will be shut down because the maximum executor failures reached. \r\n\r\n![image](https://cloud.githubusercontent.com/assets/1780746/16175374/3eb863f6-361d-11e6-9445-d8e9a0423760.png)\r\n\r\nI tracked the logged in the corresponding yarn container, and found that \"Doing rabit call after Finalize\" (see the attached log).  Then I traced the code, which is in utils:Check in rabit/src/engine.cc. It seems executors failed because the util:Check failed and return exit(-1). \r\n\r\nIt seems rabit is init and shut down in buildDistributedBoosters in train function. Is there any rabit call in prediction? \r\nAny idea about this problem? \r\n\r\n[Logs for container_1466187764481_0008_01_000007.pdf](https://github.com/dmlc/xgboost/files/322100/Logs.for.container_1466187764481_0008_01_000007.pdf)\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/comments/230472526","html_url":"https://github.com/dmlc/xgboost/issues/1284#issuecomment-230472526","issue_url":"https://api.github.com/repos/dmlc/xgboost/issues/1284","id":230472526,"user":{"login":"CodingCat","id":678008,"avatar_url":"https://avatars.githubusercontent.com/u/678008?v=3","gravatar_id":"","url":"https://api.github.com/users/CodingCat","html_url":"https://github.com/CodingCat","followers_url":"https://api.github.com/users/CodingCat/followers","following_url":"https://api.github.com/users/CodingCat/following{/other_user}","gists_url":"https://api.github.com/users/CodingCat/gists{/gist_id}","starred_url":"https://api.github.com/users/CodingCat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CodingCat/subscriptions","organizations_url":"https://api.github.com/users/CodingCat/orgs","repos_url":"https://api.github.com/users/CodingCat/repos","events_url":"https://api.github.com/users/CodingCat/events{/privacy}","received_events_url":"https://api.github.com/users/CodingCat/received_events","type":"User","site_admin":false},"created_at":"2016-07-05T13:06:10Z","updated_at":"2016-07-05T13:06:10Z","body":"![image](https://cloud.githubusercontent.com/assets/678008/16585606/b1d79b94-428f-11e6-81a0-5eed72f31e5b.png)\r\nThey are the same in implementation"}}
{"actor":{"display_login":"CodingCat"}}