{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14109","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14109/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14109/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14109/events","html_url":"https://github.com/apache/spark/pull/14109","id":164632602,"number":14109,"title":"[SPARK-16404][ML] LeastSquaresAggregators serializes unnecessary data","user":{"login":"sethah","id":7275795,"avatar_url":"https://avatars.githubusercontent.com/u/7275795?v=3","gravatar_id":"","url":"https://api.github.com/users/sethah","html_url":"https://github.com/sethah","followers_url":"https://api.github.com/users/sethah/followers","following_url":"https://api.github.com/users/sethah/following{/other_user}","gists_url":"https://api.github.com/users/sethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sethah/subscriptions","organizations_url":"https://api.github.com/users/sethah/orgs","repos_url":"https://api.github.com/users/sethah/repos","events_url":"https://api.github.com/users/sethah/events{/privacy}","received_events_url":"https://api.github.com/users/sethah/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2016-07-08T23:05:56Z","updated_at":"2016-08-05T20:19:54Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14109","html_url":"https://github.com/apache/spark/pull/14109","diff_url":"https://github.com/apache/spark/pull/14109.diff","patch_url":"https://github.com/apache/spark/pull/14109.patch"},"body":"## What changes were proposed in this pull request?\r\nSimilar to `LogisticAggregator`, `LeastSquaresAggregator` used for linear regression ends up serializing the coefficients and the features standard deviations, which is not necessary and can cause performance issues for high dimensional data. This patch removes this serialization.\r\n\r\nIn https://github.com/apache/spark/pull/13729 the approach was to pass these values directly to the add method. The approach used here, initially, is to mark these fields as transient instead which gives the benefit of keeping the signature of the add method simple and interpretable. The downside is that it requires the use of `@transient lazy val`s which are difficult to reason about if one is not quite familiar with serialization in Scala/Spark. \r\n\r\n## How was this patch tested?\r\n\r\n**MLlib**\r\n![image](https://cloud.githubusercontent.com/assets/7275795/16703660/436f79fa-4524-11e6-9022-ef00058ec718.png)\r\n\r\n**ML without patch**\r\n![image](https://cloud.githubusercontent.com/assets/7275795/16703831/c4d50b9e-4525-11e6-80cb-9b58c850cd41.png)\r\n\r\n\r\n**ML with patch**\r\n![image](https://cloud.githubusercontent.com/assets/7275795/16703675/63e0cf40-4524-11e6-9120-1f512a70e083.png)\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/237952766","html_url":"https://github.com/apache/spark/pull/14109#issuecomment-237952766","issue_url":"https://api.github.com/repos/apache/spark/issues/14109","id":237952766,"user":{"login":"sethah","id":7275795,"avatar_url":"https://avatars.githubusercontent.com/u/7275795?v=3","gravatar_id":"","url":"https://api.github.com/users/sethah","html_url":"https://github.com/sethah","followers_url":"https://api.github.com/users/sethah/followers","following_url":"https://api.github.com/users/sethah/following{/other_user}","gists_url":"https://api.github.com/users/sethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sethah/subscriptions","organizations_url":"https://api.github.com/users/sethah/orgs","repos_url":"https://api.github.com/users/sethah/repos","events_url":"https://api.github.com/users/sethah/events{/privacy}","received_events_url":"https://api.github.com/users/sethah/received_events","type":"User","site_admin":false},"created_at":"2016-08-05T20:19:54Z","updated_at":"2016-08-05T20:19:54Z","body":"@dbtsai @yanboliang I went ahead and added a couple comments so someone will not mistakenly change this behavior in the future. Let me know if you see anything else, thanks!"}}
{"actor":{"display_login":"sethah"}}