{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13729","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13729/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13729/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13729/events","html_url":"https://github.com/apache/spark/pull/13729","id":160807692,"number":13729,"title":"[SPARK-16008][ML] Remove unnecessary serialization in logistic regression","user":{"login":"sethah","id":7275795,"avatar_url":"https://avatars.githubusercontent.com/u/7275795?v=3","gravatar_id":"","url":"https://api.github.com/users/sethah","html_url":"https://github.com/sethah","followers_url":"https://api.github.com/users/sethah/followers","following_url":"https://api.github.com/users/sethah/following{/other_user}","gists_url":"https://api.github.com/users/sethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sethah/subscriptions","organizations_url":"https://api.github.com/users/sethah/orgs","repos_url":"https://api.github.com/users/sethah/repos","events_url":"https://api.github.com/users/sethah/events{/privacy}","received_events_url":"https://api.github.com/users/sethah/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-06-17T03:58:31Z","updated_at":"2016-06-17T17:06:50Z","closed_at":"2016-06-17T17:00:32Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13729","html_url":"https://github.com/apache/spark/pull/13729","diff_url":"https://github.com/apache/spark/pull/13729.diff","patch_url":"https://github.com/apache/spark/pull/13729.patch"},"body":"JIRA: [SPARK-16008](https://issues.apache.org/jira/browse/SPARK-16008)\r\n\r\n## What changes were proposed in this pull request?\r\n`LogisticAggregator` stores references to two arrays of dimension `numFeatures` which are serialized before the combine op, unnecessarily. This results in the shuffle write being ~3x (for multiclass logistic regression, this number will go up) larger than it should be (in MLlib, for instance, it is 3x smaller).\r\n\r\nThis patch modifies `LogisticAggregator.add` to accept the two arrays as method parameters which avoids the serialization. \r\n\r\n## How was this patch tested?\r\n\r\nI tested this locally and verified the serialization reduction. \r\n\r\n![image](https://cloud.githubusercontent.com/assets/7275795/16140387/d2974bac-3404-11e6-94f9-268860c931a2.png)\r\n\r\nAdditionally, I ran some tests of a 4 node cluster (4x48 cores, 4x128 GB RAM). Data set size of 2M rows and 10k features showed >2x iteration speedup. \r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/226825695","html_url":"https://github.com/apache/spark/pull/13729#issuecomment-226825695","issue_url":"https://api.github.com/repos/apache/spark/issues/13729","id":226825695,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-06-17T17:06:50Z","updated_at":"2016-06-17T17:06:50Z","body":"**[Test build #60712 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60712/consoleFull)** for PR 13729 at commit [`5d668a6`](https://github.com/apache/spark/commit/5d668a6f93859801262393540fe954257f433a35).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}