{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22954","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22954/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22954/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22954/events","html_url":"https://github.com/apache/spark/pull/22954","id":377765075,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI4NjI4NTAx","number":22954,"title":"[SPARK-25981][R] Enables Arrow optimization from R DataFrame to Spark DataFrame","user":{"login":"HyukjinKwon","id":6477701,"node_id":"MDQ6VXNlcjY0Nzc3MDE=","avatar_url":"https://avatars0.githubusercontent.com/u/6477701?v=4","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":32,"created_at":"2018-11-06T09:45:02Z","updated_at":"2018-11-13T12:05:27Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22954","html_url":"https://github.com/apache/spark/pull/22954","diff_url":"https://github.com/apache/spark/pull/22954.diff","patch_url":"https://github.com/apache/spark/pull/22954.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR  targets to support Arrow optimization for conversion from R DataFrame to Spark DataFrame.\r\nLike PySpark side, it falls back to non-optimization code path when it's unable to use Arrow optimization.\r\n\r\nThis can be tested as below:\r\n\r\n```bash\r\n$ ./bin/sparkR --conf spark.sql.execution.arrow.enabled=true\r\n```\r\n\r\n```r\r\ncollect(createDataFrame(mtcars))\r\n```\r\n\r\n### Requirements (to be documented):\r\n  - R 3.5.x \r\n  - Arrow package 0.12+ (not released yet)\r\n  - Arrow CRAN release (ARROW-3204)\r\n  - withr package\r\n\r\n### Benchmarks:\r\n\r\n**Shall:**\r\n\r\n```bash\r\nsync && sudo purge\r\n./bin/sparkR --conf spark.sql.execution.arrow.enabled=false\r\n```\r\n\r\n```bash\r\nsync && sudo purge\r\n./bin/sparkR --conf spark.sql.execution.arrow.enabled=true\r\n```\r\n\r\n**R code:**\r\n\r\n```r\r\ncreateDataFrame(mtcars) # Initializes\r\nrdf <- read.csv(\"500000.csv\")\r\n\r\ntest <- function() {\r\n  options(digits.secs = 6) # milliseconds\r\n  start.time <- Sys.time()\r\n  createDataFrame(rdf)\r\n  end.time <- Sys.time()\r\n  time.taken <- end.time - start.time\r\n  print(time.taken)\r\n}\r\n\r\ntest()\r\n```\r\n\r\n**Data (350 MB):**\r\n\r\n```r\r\nobject.size(read.csv(\"500000.csv\"))\r\n350379504 bytes\r\n```\r\n\r\n\"500000 Records\"  http://eforexcel.com/wp/downloads-16-sample-csv-files-data-sets-for-testing/\r\n\r\n**Results:**\r\n\r\n```\r\nTime difference of 28.3867 secs\r\n```\r\n\r\n```\r\nTime difference of 2.969984 secs\r\n```\r\n\r\nThe performance improvement was **955%**.\r\n\r\n### Limitations:\r\n\r\nFor now, Arrow optimization with R does not support when the data is `POSIXct` (ARROW-3614) and `raw`, and when user explicitly gives float type in the schema. They produce corrupt values.\r\nIn this case, we decide to fall back to non-optimization code path.\r\n\r\n## How was this patch tested?\r\n\r\nSmall test was added.\r\n\r\nI manually forced to set this optimization `true` for _all_ R tests and they were _all_ passed (with few of fallback warnings).\r\n\r\n**TODOs:**\r\n- [x] Draft codes\r\n- [x] make the tests passed\r\n- [x] make the CRAN check pass\r\n- [x] Performance measurement\r\n- [x] Supportability investigation (for instance types)\r\n- [ ] Wait for Arrow 0.12.0 release\r\n- [ ] Fix and match it to Arrow 0.12.0\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/438243068","html_url":"https://github.com/apache/spark/pull/22954#issuecomment-438243068","issue_url":"https://api.github.com/repos/apache/spark/issues/22954","id":438243068,"node_id":"MDEyOklzc3VlQ29tbWVudDQzODI0MzA2OA==","user":{"login":"SparkQA","id":8108735,"node_id":"MDQ6VXNlcjgxMDg3MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2018-11-13T12:05:27Z","updated_at":"2018-11-13T12:05:27Z","author_association":"NONE","body":"**[Test build #98760 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98760/testReport)** for PR 22954 at commit [`954bc0e`](https://github.com/apache/spark/commit/954bc0eec206902cb8176338e1f72886f5b3c626).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}