{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13459","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13459/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13459/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13459/events","html_url":"https://github.com/apache/spark/pull/13459","id":158050938,"number":13459,"title":"[SPARK-15726] [SQL] Make DatasetBenchmark fairer among Dataset, DataFrame and RDD","user":{"login":"inouehrs","id":6196943,"avatar_url":"https://avatars.githubusercontent.com/u/6196943?v=3","gravatar_id":"","url":"https://api.github.com/users/inouehrs","html_url":"https://github.com/inouehrs","followers_url":"https://api.github.com/users/inouehrs/followers","following_url":"https://api.github.com/users/inouehrs/following{/other_user}","gists_url":"https://api.github.com/users/inouehrs/gists{/gist_id}","starred_url":"https://api.github.com/users/inouehrs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/inouehrs/subscriptions","organizations_url":"https://api.github.com/users/inouehrs/orgs","repos_url":"https://api.github.com/users/inouehrs/repos","events_url":"https://api.github.com/users/inouehrs/events{/privacy}","received_events_url":"https://api.github.com/users/inouehrs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2016-06-02T03:11:51Z","updated_at":"2016-08-05T04:12:37Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13459","html_url":"https://github.com/apache/spark/pull/13459","diff_url":"https://github.com/apache/spark/pull/13459.diff","patch_url":"https://github.com/apache/spark/pull/13459.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nDatasetBenchmark compares the performances of RDD, DataFrame and Dataset while running the same operations.\r\nIn backToBackMap test case, however, only DataFrame implementation executes less work compared to RDD or Dataset implementations. This test case processes Long+String pairs, but the output from the DataFrame implementation does not include String part while RDD or Dataset generates Long+String pairs as output. This difference significantly changes the performance characteristics due to the String manipulation and creation overheads. After the fix RDD outperforms DataFrame, while DataFrame was more than 2x faster than RDD without the fix. Also, the performance gap between DataFrame and Dataset becomes much narrower.\r\n\r\nOf course, this issue does not affect Spark users, but it may confuse Spark developers.\r\n\r\n```scala\r\n// DataFrame\r\nval df = spark.range(1, numRows).select($\"id\".as(\"l\"), $\"id\".cast(StringType).as(\"s\"))\r\nvar res = df\r\nres = res.select($\"l\" + 1 as \"l\")\r\n// this should be res = res.select($\"l\" + 1 as \"l\", $\"s\") for fair comparison\r\n\r\n// Dataset \r\ncase class Data(l: Long, s: String)\r\nval func = (d: Data) => Data(d.l + 1, d.s)\r\nvar res = df.as[Data]\r\nres = res.map(func)\r\n```\r\n\r\nAdditionally, I added a new test case named \"back-to-back map for primitive\". This is almost equivalent with the old behavior of the DataFrame implementation of back-to-back map.\r\n\r\n```\r\nwithout fix\r\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 3.10.0-229.el7.x86_64\r\nIntel Xeon E3-12xx v2 (Ivy Bridge)\r\nback-to-back map:                        Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nRDD                                           2051 / 2077         48.7          20.5       1.0X\r\nDataFrame                                      755 /  940        132.5           7.5       2.7X\r\nDataset                                       6155 / 6680         16.2          61.6       0.3X\r\n\r\nwith fix\r\nback-to-back map:                        Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nRDD                                           2077 / 2259         48.1          20.8       1.0X\r\nDataFrame                                     3030 / 3310         33.0          30.3       0.7X\r\nDataset                                       6504 / 7006         15.4          65.0       0.3X\r\n\r\nback-to-back map for primitive:          Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nRDD                                           1073 / 1509         93.2          10.7       1.0X\r\nDataFrame                                      763 /  913        131.0           7.6       1.4X\r\nDataset                                       4189 / 4312         23.9          41.9       0.3X\r\n```\r\n\r\nNote that DatasetBenchmark causes JVM crash in an aggregate test case. This is not related to this issue.\r\nI already created a jira entry and submited a pull request for the aggregate issue.\r\nhttps://issues.apache.org/jira/browse/SPARK-15704\r\nhttps://github.com/apache/spark/pull/13446\r\n\r\n\r\n## How was this patch tested?\r\nBy executing the DatasetBenchmark\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/237750973","html_url":"https://github.com/apache/spark/pull/13459#issuecomment-237750973","issue_url":"https://api.github.com/repos/apache/spark/issues/13459","id":237750973,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-08-05T04:12:37Z","updated_at":"2016-08-05T04:12:37Z","body":"**[Test build #63253 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63253/consoleFull)** for PR 13459 at commit [`1866ae4`](https://github.com/apache/spark/commit/1866ae4aab89e29577b27f38315ff26bd3bb636c)."}}
{"actor":{"display_login":"SparkQA"}}