{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/3470","repository_url":"https://api.github.com/repos/dmlc/xgboost","labels_url":"https://api.github.com/repos/dmlc/xgboost/issues/3470/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/xgboost/issues/3470/comments","events_url":"https://api.github.com/repos/dmlc/xgboost/issues/3470/events","html_url":"https://github.com/dmlc/xgboost/issues/3470","id":341319900,"node_id":"MDU6SXNzdWUzNDEzMTk5MDA=","number":3470,"title":"XGBoost in Spark \"stops\" for a couple hours, then continues","user":{"login":"olbapjose","id":34943494,"node_id":"MDQ6VXNlcjM0OTQzNDk0","avatar_url":"https://avatars0.githubusercontent.com/u/34943494?v=4","gravatar_id":"","url":"https://api.github.com/users/olbapjose","html_url":"https://github.com/olbapjose","followers_url":"https://api.github.com/users/olbapjose/followers","following_url":"https://api.github.com/users/olbapjose/following{/other_user}","gists_url":"https://api.github.com/users/olbapjose/gists{/gist_id}","starred_url":"https://api.github.com/users/olbapjose/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olbapjose/subscriptions","organizations_url":"https://api.github.com/users/olbapjose/orgs","repos_url":"https://api.github.com/users/olbapjose/repos","events_url":"https://api.github.com/users/olbapjose/events{/privacy}","received_events_url":"https://api.github.com/users/olbapjose/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-07-15T13:09:55Z","updated_at":"2018-07-18T02:11:52Z","closed_at":null,"author_association":"NONE","body":"Spark version 2.2.0. \r\nXGBoost version: 0.80-SNAPSHOT\r\nCluster with CentOS, 3 workers, 90 GB RAM.\r\n\r\nApparently, when training under Spark (Scala), I can see the Spark logs when each tree is fit, something like: \r\n\r\n[14:20:03] /home/myuser/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 38122 extra nodes, 0 pruned nodes, max_depth=15\r\n\r\nI can see a new tree after every few seconds but at some point, sometimes after one hour or two, I see no new messages when refreshing the log. Then after two or three hours \"in pause\", the process continues and I can see new trees again added to the log. This does not always happen but it does happen quite often.\r\n\r\nIs there any explanation for this? \r\n\r\nThanks\r\n\r\n**EDIT:** I thought it could be related to memory space but it seems it is not as there's much free memory right now:\r\n\r\n![image](https://user-images.githubusercontent.com/34943494/42734354-cdfa8eee-8842-11e8-8528-055914d3d40b.png)\r\n"},"comment":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/comments/405786107","html_url":"https://github.com/dmlc/xgboost/issues/3470#issuecomment-405786107","issue_url":"https://api.github.com/repos/dmlc/xgboost/issues/3470","id":405786107,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTc4NjEwNw==","user":{"login":"CodingCat","id":678008,"node_id":"MDQ6VXNlcjY3ODAwOA==","avatar_url":"https://avatars1.githubusercontent.com/u/678008?v=4","gravatar_id":"","url":"https://api.github.com/users/CodingCat","html_url":"https://github.com/CodingCat","followers_url":"https://api.github.com/users/CodingCat/followers","following_url":"https://api.github.com/users/CodingCat/following{/other_user}","gists_url":"https://api.github.com/users/CodingCat/gists{/gist_id}","starred_url":"https://api.github.com/users/CodingCat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CodingCat/subscriptions","organizations_url":"https://api.github.com/users/CodingCat/orgs","repos_url":"https://api.github.com/users/CodingCat/repos","events_url":"https://api.github.com/users/CodingCat/events{/privacy}","received_events_url":"https://api.github.com/users/CodingCat/received_events","type":"User","site_admin":false},"created_at":"2018-07-18T02:11:52Z","updated_at":"2018-07-18T02:11:52Z","author_association":"MEMBER","body":"most of memory usage in training set is in native memory space, so it will not be displayed in Spark UI\r\n\r\nif you are using YARN and you are sure that your on-heap space is lightly used, you can try to set spark.executor.memoryOverhead larger to allocate more for XGBoost"}}
{"actor":{"display_login":"CodingCat"}}