{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16117","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16117/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16117/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16117/events","html_url":"https://github.com/apache/spark/pull/16117","id":193058994,"number":16117,"title":"[SPARK-18686][SparkR][ML] Several cleanup and improvements for spark.logit.","user":{"login":"yanboliang","id":1962026,"avatar_url":"https://avatars.githubusercontent.com/u/1962026?v=3","gravatar_id":"","url":"https://api.github.com/users/yanboliang","html_url":"https://github.com/yanboliang","followers_url":"https://api.github.com/users/yanboliang/followers","following_url":"https://api.github.com/users/yanboliang/following{/other_user}","gists_url":"https://api.github.com/users/yanboliang/gists{/gist_id}","starred_url":"https://api.github.com/users/yanboliang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yanboliang/subscriptions","organizations_url":"https://api.github.com/users/yanboliang/orgs","repos_url":"https://api.github.com/users/yanboliang/repos","events_url":"https://api.github.com/users/yanboliang/events{/privacy}","received_events_url":"https://api.github.com/users/yanboliang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-12-02T08:21:44Z","updated_at":"2016-12-07T08:31:09Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16117","html_url":"https://github.com/apache/spark/pull/16117","diff_url":"https://github.com/apache/spark/pull/16117.diff","patch_url":"https://github.com/apache/spark/pull/16117.patch"},"body":"## What changes were proposed in this pull request?\r\nSeveral cleanup and improvements for ```spark.logit```:\r\n* ```summary``` should return coefficients matrix, and should output labels for each class if the model is multinomial logistic regression model.\r\n* ```summary``` should not return ```areaUnderROC, roc, pr, ...```, since most of them are DataFrame which are less important for R users. Meanwhile, these metrics ignore instance weights (setting all to 1.0) which will be changed in later Spark version. In case it will introduce breaking changes, we do not expose them currently.\r\n* SparkR test improvement: comparing the training result with native R glmnet.\r\n* Remove argument ```aggregationDepth``` from ```spark.logit```, since it's an expert Param(related with Spark architecture and job execution) that would be used rarely by R users.\r\n\r\n## How was this patch tested?\r\nUnit tests.\r\n\r\nThe ```summary``` output after this change:\r\nmultinomial logistic regression:\r\n```\r\n> df <- suppressWarnings(createDataFrame(iris))\r\n> model <- spark.logit(df, Species ~ ., regParam = 0.5)\r\n> summary(model)\r\n$coefficients\r\n             versicolor  virginica   setosa\r\n(Intercept)  1.514031    -2.609108   1.095077\r\nSepal_Length 0.02511006  0.2649821   -0.2900921\r\nSepal_Width  -0.5291215  -0.02016446 0.549286\r\nPetal_Length 0.03647411  0.1544119   -0.190886\r\nPetal_Width  0.000236092 0.4195804   -0.4198165\r\n```\r\nbinomial logistic regression:\r\n```\r\n> df <- suppressWarnings(createDataFrame(iris))\r\n> training <- df[df$Species %in% c(\"versicolor\", \"virginica\"), ]\r\n> model <- spark.logit(training, Species ~ ., regParam = 0.5)\r\n> summary(model)\r\n$coefficients\r\n             Estimate\r\n(Intercept)  -6.053815\r\nSepal_Length 0.2449379\r\nSepal_Width  0.1648321\r\nPetal_Length 0.4730718\r\nPetal_Width  1.031947\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/265386834","html_url":"https://github.com/apache/spark/pull/16117#issuecomment-265386834","issue_url":"https://api.github.com/repos/apache/spark/issues/16117","id":265386834,"user":{"login":"yanboliang","id":1962026,"avatar_url":"https://avatars.githubusercontent.com/u/1962026?v=3","gravatar_id":"","url":"https://api.github.com/users/yanboliang","html_url":"https://github.com/yanboliang","followers_url":"https://api.github.com/users/yanboliang/followers","following_url":"https://api.github.com/users/yanboliang/following{/other_user}","gists_url":"https://api.github.com/users/yanboliang/gists{/gist_id}","starred_url":"https://api.github.com/users/yanboliang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yanboliang/subscriptions","organizations_url":"https://api.github.com/users/yanboliang/orgs","repos_url":"https://api.github.com/users/yanboliang/repos","events_url":"https://api.github.com/users/yanboliang/events{/privacy}","received_events_url":"https://api.github.com/users/yanboliang/received_events","type":"User","site_admin":false},"created_at":"2016-12-07T08:31:09Z","updated_at":"2016-12-07T08:31:09Z","body":"Since this will involve breaking change if we merge it after 2.1 release, and actually I think it belongs to the scope of QA, so I'll merge it into master and branch-2.1, thanks for all your reviewing."}}
{"actor":{"display_login":"yanboliang"}}