{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15375","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15375/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15375/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15375/events","html_url":"https://github.com/apache/spark/pull/15375","id":181338110,"number":15375,"title":"[SPARK-17790][SPARKR] Support for parallelizing R data.frame larger than 2GB","user":{"login":"falaki","id":512364,"avatar_url":"https://avatars.githubusercontent.com/u/512364?v=3","gravatar_id":"","url":"https://api.github.com/users/falaki","html_url":"https://github.com/falaki","followers_url":"https://api.github.com/users/falaki/followers","following_url":"https://api.github.com/users/falaki/following{/other_user}","gists_url":"https://api.github.com/users/falaki/gists{/gist_id}","starred_url":"https://api.github.com/users/falaki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/falaki/subscriptions","organizations_url":"https://api.github.com/users/falaki/orgs","repos_url":"https://api.github.com/users/falaki/repos","events_url":"https://api.github.com/users/falaki/events{/privacy}","received_events_url":"https://api.github.com/users/falaki/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2016-10-06T06:58:25Z","updated_at":"2016-10-11T17:42:31Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15375","html_url":"https://github.com/apache/spark/pull/15375","diff_url":"https://github.com/apache/spark/pull/15375.diff","patch_url":"https://github.com/apache/spark/pull/15375.patch"},"body":"## What changes were proposed in this pull request?\r\nIf the R data structure that is being parallelized is larger than `INT_MAX` we use files to transfer data to JVM. The serialization protocol mimics Python pickling. This allows us to simply call `PythonRDD.readRDDFromFile` to create the RDD.\r\n\r\nI tested this on my MacBook. Following code works with this patch:\r\n```R\r\nintMax <- .Machine$integer.max\r\nlargeVec <- 1:intMax\r\nrdd <- SparkR:::parallelize(sc, largeVec, 2)\r\n```\r\n\r\n## How was this patch tested?\r\n* [x] Unit tests\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/252990210","html_url":"https://github.com/apache/spark/pull/15375#issuecomment-252990210","issue_url":"https://api.github.com/repos/apache/spark/issues/15375","id":252990210,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-10-11T17:42:31Z","updated_at":"2016-10-11T17:42:31Z","body":"**[Test build #66750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66750/consoleFull)** for PR 15375 at commit [`836e874`](https://github.com/apache/spark/commit/836e8745c346c59f78958e10aec1c6f9537242b9)."}}
{"actor":{"display_login":"SparkQA"}}