{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10788","labels_url":"https://api.github.com/repos/apache/spark/issues/10788/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10788/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10788/events","html_url":"https://github.com/apache/spark/pull/10788","id":127037952,"number":10788,"title":"[SPARK-7780][MLLIB] intercept in logisticregressionwith lbfgs should not be regularized","user":{"login":"holdenk","id":59893,"avatar_url":"https://avatars.githubusercontent.com/u/59893?v=3","gravatar_id":"","url":"https://api.github.com/users/holdenk","html_url":"https://github.com/holdenk","followers_url":"https://api.github.com/users/holdenk/followers","following_url":"https://api.github.com/users/holdenk/following{/other_user}","gists_url":"https://api.github.com/users/holdenk/gists{/gist_id}","starred_url":"https://api.github.com/users/holdenk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/holdenk/subscriptions","organizations_url":"https://api.github.com/users/holdenk/orgs","repos_url":"https://api.github.com/users/holdenk/repos","events_url":"https://api.github.com/users/holdenk/events{/privacy}","received_events_url":"https://api.github.com/users/holdenk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2016-01-16T17:10:30Z","updated_at":"2016-01-25T22:03:41Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10788","html_url":"https://github.com/apache/spark/pull/10788","diff_url":"https://github.com/apache/spark/pull/10788.diff","patch_url":"https://github.com/apache/spark/pull/10788.patch"},"body":"The intercept in Logistic Regression represents a prior on categories which should not be regularized. In MLlib, the regularization is handled through Updater, and the Updater penalizes all the components without excluding the intercept which resulting poor training accuracy with regularization.\r\nThe new implementation in ML framework handles this properly, and we should call the implementation in ML from MLlib since majority of users are still using MLlib api.\r\nNote that both of them are doing feature scalings to improve the convergence, and the only difference is ML version doesn't regularize the intercept. As a result, when lambda is zero, they will converge to the same solution.\r\n\r\nPreviously partially reviewed at https://github.com/apache/spark/pull/6386#issuecomment-168781424 re-opening for @dbtsai to review."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/174688059","html_url":"https://github.com/apache/spark/pull/10788#issuecomment-174688059","issue_url":"https://api.github.com/repos/apache/spark/issues/10788","id":174688059,"user":{"login":"holdenk","id":59893,"avatar_url":"https://avatars.githubusercontent.com/u/59893?v=3","gravatar_id":"","url":"https://api.github.com/users/holdenk","html_url":"https://github.com/holdenk","followers_url":"https://api.github.com/users/holdenk/followers","following_url":"https://api.github.com/users/holdenk/following{/other_user}","gists_url":"https://api.github.com/users/holdenk/gists{/gist_id}","starred_url":"https://api.github.com/users/holdenk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/holdenk/subscriptions","organizations_url":"https://api.github.com/users/holdenk/orgs","repos_url":"https://api.github.com/users/holdenk/repos","events_url":"https://api.github.com/users/holdenk/events{/privacy}","received_events_url":"https://api.github.com/users/holdenk/received_events","type":"User","site_admin":false},"created_at":"2016-01-25T22:03:41Z","updated_at":"2016-01-25T22:03:41Z","body":"Jenkins retest this please"}}