{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14359","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14359/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14359/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14359/events","html_url":"https://github.com/apache/spark/pull/14359","id":167507126,"number":14359,"title":"[SPARK-16719][ML] Random Forests should communicate fewer trees on each iteration","user":{"login":"jkbradley","id":5084283,"avatar_url":"https://avatars.githubusercontent.com/u/5084283?v=3","gravatar_id":"","url":"https://api.github.com/users/jkbradley","html_url":"https://github.com/jkbradley","followers_url":"https://api.github.com/users/jkbradley/followers","following_url":"https://api.github.com/users/jkbradley/following{/other_user}","gists_url":"https://api.github.com/users/jkbradley/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbradley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbradley/subscriptions","organizations_url":"https://api.github.com/users/jkbradley/orgs","repos_url":"https://api.github.com/users/jkbradley/repos","events_url":"https://api.github.com/users/jkbradley/events{/privacy}","received_events_url":"https://api.github.com/users/jkbradley/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-26T02:21:21Z","updated_at":"2016-07-26T02:22:53Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14359","html_url":"https://github.com/apache/spark/pull/14359","diff_url":"https://github.com/apache/spark/pull/14359.diff","patch_url":"https://github.com/apache/spark/pull/14359.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nRandomForest currently sends the entire forest to each worker on each iteration. This is because (a) the node queue is FIFO and (b) the closure references the entire array of trees (topNodes). (a) causes RFs to handle splits in many trees, especially early on in learning. (b) sends all trees explicitly.\r\n\r\nThis PR:\r\n(a) Change the RF node queue to be FILO, so that RFs tend to focus on 1 or a few trees before focusing on others.\r\n(b) Change topNodes to pass only the trees required on that iteration.\r\n\r\n## How was this patch tested?\r\n\r\nUnit tests:\r\n* Existing tests for correctness of tree learning\r\n* New test: NodeQueue should be FILO\r\n* Manually modifying code and running tests to verify that a small number of trees are communicated on each iteration\r\n  * This last item is hard to test via unit tests given the current APIs."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/235144392","html_url":"https://github.com/apache/spark/pull/14359#issuecomment-235144392","issue_url":"https://api.github.com/repos/apache/spark/issues/14359","id":235144392,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-07-26T02:22:53Z","updated_at":"2016-07-26T02:22:53Z","body":"**[Test build #62858 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62858/consoleFull)** for PR 14359 at commit [`6fcfb4b`](https://github.com/apache/spark/commit/6fcfb4b0e158ba86371ad4d0728490f3a8e7caeb)."}}
{"actor":{"display_login":"SparkQA"}}