{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13403","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13403/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13403/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13403/events","html_url":"https://github.com/apache/spark/pull/13403","id":157592121,"number":13403,"title":"[SPARK-15660][CORE] RDD and Dataset should show the consistent values for variance/stdev.","user":{"login":"dongjoon-hyun","id":9700541,"avatar_url":"https://avatars.githubusercontent.com/u/9700541?v=3","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2016-05-31T05:25:14Z","updated_at":"2016-06-09T08:10:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13403","html_url":"https://github.com/apache/spark/pull/13403","diff_url":"https://github.com/apache/spark/pull/13403.diff","patch_url":"https://github.com/apache/spark/pull/13403.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIn Spark-11490, `variance/stdev` are redefined as the **sample** `variance/stdev` instead of population ones. This PR addresses the only remaining legacy in RDD. This may cause breaking changes, but we had better be consistent in Spark 2.0 if possible. This PR also `popVariance` and `popStdev` functions.\r\n\r\n**Before**\r\n```scala\r\nscala> val ds = spark.createDataset(Seq(1.0, 2.0, 3.0))\r\nds: org.apache.spark.sql.Dataset[Double] = [value: double]\r\n\r\nscala> ds.describe().show()\r\n+-------+-----+                                                                 \r\n|summary|value|\r\n+-------+-----+\r\n|  count|    3|\r\n|   mean|  2.0|\r\n| stddev|  1.0|\r\n|    min|  1.0|\r\n|    max|  3.0|\r\n+-------+-----+\r\n\r\nscala> ds.rdd.stdev\r\nres1: Double = 0.816496580927726\r\n```\r\n\r\n**After**\r\n```scala\r\nscala> ds.rdd.stdev\r\nres0: Double = 1.0\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nPass the updated Jenkins tests."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/224827848","html_url":"https://github.com/apache/spark/pull/13403#issuecomment-224827848","issue_url":"https://api.github.com/repos/apache/spark/issues/13403","id":224827848,"user":{"login":"dongjoon-hyun","id":9700541,"avatar_url":"https://avatars.githubusercontent.com/u/9700541?v=3","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"created_at":"2016-06-09T08:10:33Z","updated_at":"2016-06-09T08:10:33Z","body":"Although we can not change old API, I think it's a good idea to add `popVariance` and `popStdev` clearly.\r\n\r\nIf everything in this PR is now allowed, what about just adding an explicit note on old `StatCounter.variance` and `StatCounter.stdev`?\r\n\r\nhttp://spark.apache.org/docs/2.0.0-preview/api/scala/index.html#org.apache.spark.util.StatCounter\r\n\r\n"}}
{"actor":{"display_login":"dongjoon-hyun"}}