{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16149","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16149/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16149/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16149/events","html_url":"https://github.com/apache/spark/pull/16149","id":193577202,"number":16149,"title":"[SPARK-18715][ML]Fix AIC calculations in Binomial GLM","user":{"login":"actuaryzhang","id":11082368,"avatar_url":"https://avatars.githubusercontent.com/u/11082368?v=3","gravatar_id":"","url":"https://api.github.com/users/actuaryzhang","html_url":"https://github.com/actuaryzhang","followers_url":"https://api.github.com/users/actuaryzhang/followers","following_url":"https://api.github.com/users/actuaryzhang/following{/other_user}","gists_url":"https://api.github.com/users/actuaryzhang/gists{/gist_id}","starred_url":"https://api.github.com/users/actuaryzhang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/actuaryzhang/subscriptions","organizations_url":"https://api.github.com/users/actuaryzhang/orgs","repos_url":"https://api.github.com/users/actuaryzhang/repos","events_url":"https://api.github.com/users/actuaryzhang/events{/privacy}","received_events_url":"https://api.github.com/users/actuaryzhang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2016-12-05T18:41:10Z","updated_at":"2016-12-13T09:52:34Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16149","html_url":"https://github.com/apache/spark/pull/16149","diff_url":"https://github.com/apache/spark/pull/16149.diff","patch_url":"https://github.com/apache/spark/pull/16149.patch"},"body":"The AIC calculation in Binomial GLM seems to be off when the response or weight is non-integer: the result is different from that in R. This issue arises when one models rates, i.e, num of successes normalized over num of trials, and uses num of trials as weights. In this case, the effective likelihood is  weight * label ~ binomial(weight, mu), where weight = number of trials, and weight * label = number of successes and mu = is the success rate.  \r\n\r\n@srowen @sethah @yanboliang @HyukjinKwon @zhengruifeng \r\n\r\n## What changes were proposed in this pull request?\r\nI suggest changing the current aic calculation for the Binomial family from \r\n```\r\n-2.0 * predictions.map { case (y: Double, mu: Double, weight: Double) =>\r\n        weight * dist.Binomial(1, mu).logProbabilityOf(math.round(y).toInt)\r\n      }.sum()\r\n```\r\nto the following which generalizes to the case of real-valued response and weights. \r\n```\r\n      -2.0 * predictions.map { case (y: Double, mu: Double, weight: Double) =>\r\n        val wt = math.round(weight).toInt\r\n        if (wt == 0){\r\n          0.0\r\n        } else {\r\n          dist.Binomial(wt, mu).logProbabilityOf(math.round(y * weight).toInt)\r\n        }\r\n      }.sum()\r\n``` \r\n## How was this patch tested?\r\nI will write the unit test once the community wants to include the proposed change. For now, the following modifies existing tests in weighted Binomial GLM to illustrate the issue. The second label is changed from 0 to 0.5. \r\n\r\n```\r\nval datasetWithWeight = Seq(\r\n    (1.0, 1.0, 0.0, 5.0),\r\n    (0.5, 2.0, 1.0, 2.0),\r\n    (1.0, 3.0, 2.0, 1.0),\r\n    (0.0, 4.0, 3.0, 3.0)\r\n  ).toDF(\"y\", \"w\", \"x1\", \"x2\")\r\n\r\nval formula = (new RFormula()\r\n  .setFormula(\"y ~ x1 + x2\")\r\n  .setFeaturesCol(\"features\")\r\n  .setLabelCol(\"label\"))\r\nval output = formula.fit(datasetWithWeight).transform(datasetWithWeight).select(\"features\", \"label\", \"w\")\r\n\r\nval glr = new GeneralizedLinearRegression()\r\n    .setFamily(\"binomial\")\r\n    .setWeightCol(\"w\")\r\n    .setFitIntercept(false)\r\n    .setRegParam(0)\r\n\r\nval model = glr.fit(output)\r\nmodel.summary.aic\r\n```\r\nThe AIC from Spark is 17.3227, and the AIC from R is 15.66454. "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/266694559","html_url":"https://github.com/apache/spark/pull/16149#issuecomment-266694559","issue_url":"https://api.github.com/repos/apache/spark/issues/16149","id":266694559,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-12-13T09:52:34Z","updated_at":"2016-12-13T09:52:34Z","body":"**[Test build #3495 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3495/consoleFull)** for PR 16149 at commit [`6e6c48b`](https://github.com/apache/spark/commit/6e6c48b79065666e1e896eec76e1ffa8cb751b6e)."}}
{"actor":{"display_login":"SparkQA"}}