title: How to load large R data.frames into Spark using SparkR's as.DataFrame()?
body: <p>My goal is to load large R data.frame into Spark. The size of the data.frame is 5 mil. rows and 7 columns of various types. Once loaded into R, this data.frame takes up approx. 200mb of memory. However, when I try loading it into Spark using <code>as.DataFrame()</code> function, the R session gets occupied forever, it has been running for 1 hour, and I had to cancel the operation. </p>
  
  <p>Here are the details:</p>
  
  <p>I am creating the following dataset to using in this example:</p>
  
  <pre><code>n=5e6 # set sample size
  
  d &lt;- data.frame(
      v1=base::sample(1:9,n,replace=TRUE), 
      v2=base::sample(1000:9000,n,replace=TRUE), 
      v3=seq(as.Date("2016-01-01"), as.Date("2016-12-31"), by = "day")[base::sample(1:365,n,replace=TRUE)],
      v4=LETTERS[base::sample(1:length(LETTERS),n,replace=TRUE)], 
      v5=base::sample(1000:9000,n,replace=TRUE)/1000, 
      v6=seq(ISOdate(2016,1,1), ISOdate(2018,1,1), "sec")[base::sample(1:63158401,n,replace=TRUE)],
      v7=c(TRUE,FALSE)[base::sample(1:2,n,replace=TRUE)]
  )
  </code></pre>
  
  <p>The above creates a sample data.frame</p>
  
  <p>Size, approx 200mb:</p>
  
  <pre><code>paste0("size: ", round(as.numeric(object.size(d))/1000000,1)," mb")
  </code></pre>
  
  <p>Next, I create a Spark session:</p>
  
  <pre><code>Sys.setenv(SPARK_HOME='C:\\soft\\spark-2.0.0-bin-hadoop2.7',HADOOP_HOME='C:\\soft\\hadoop')
  .libPaths(c(file.path(Sys.getenv('SPARK_HOME'), 'R', 'lib'),.libPaths()))
  Sys.setenv('SPARKR_SUBMIT_ARGS'='"sparkr-shell"')
  
  library(SparkR)
  library(rJava)
  sparkR.session(enableHiveSupport = FALSE,master = "local[*]", sparkConfig = list(spark.driver.memory = "1g",spark.sql.warehouse.dir="C:\\soft\\hadoop\\bin"))
  </code></pre>
  
  <p>Now, I am trying to load the data.frame I created above into Spark:</p>
  
  <pre><code>d_sd &lt;- as.DataFrame(d)
  </code></pre>
  
  <p>The above command takes forever to run. </p>
  
  <p>Is there something wrong I am doing?
  Can it be related to the class() of columns in my original R data.frame?
  Should I take an alternative approach to loading large datasets from R into Spark? If yes, please feel free to suggest something.</p>
  
  <p>Thank you in advance.</p>
  
  <p>PS:</p>
  
  <p>I am able to quickly convert and manipulate small datasets in Spark using this method.</p>
  
  <p>Here is some background information on my R session and OS I am running:</p>
  
  <p>R version 3.2.5 (2016-04-14)
  Platform: x86_64-w64-mingw32/x64 (64-bit)
  Running under: Windows 7 x64 (build 7601) Service Pack 1</p>
  
  <p>I am running Microsoft version of R (Revolution) under Windows 7 Professional (64 bit), 8 GB of RAM. Processor: i5-2520M @ 2.50GHz</p>
  
  <hr>
  
  <p><strong>EDIT 2016-09-19:</strong></p>
  
  <p>Thank you, Zeydy Ortiz and Mohit Bansal. Based on your answers, I have tried the following, but I am still facing the  same issue:</p>
  
  <pre><code>Sys.setenv(SPARK_HOME='C:\\soft\\spark-2.0.0-bin-hadoop2.7',HADOOP_HOME='C:\\soft\\hadoop')
  .libPaths(c(file.path(Sys.getenv('SPARK_HOME'), 'R', 'lib'),.libPaths()))
  Sys.setenv('SPARKR_SUBMIT_ARGS'='"sparkr-shell"')
  
  library(SparkR)
  library(rJava)
  sparkR.session(enableHiveSupport = FALSE,master = "local[*]", sparkConfig = list(spark.driver.memory = "1g",spark.sql.warehouse.dir="C:\\soft\\hadoop\\bin"))
  
  
  n=5e6 # set sample size
  
  d_sd &lt;- createDataFrame(sqlContext,data=data.frame(
          v1=base::sample(1:9,n,replace=TRUE), 
          v2=base::sample(1000:9000,n,replace=TRUE), 
          v3=seq(as.Date("2016-01-01"), as.Date("2016-12-31"), by = "day")[base::sample(1:365,n,replace=TRUE)],
          v4=LETTERS[base::sample(1:length(LETTERS),n,replace=TRUE)], 
          v5=base::sample(1000:9000,n,replace=TRUE)/1000, 
          v6=seq(ISOdate(2016,1,1), ISOdate(2018,1,1), "sec")[base::sample(1:63158401,n,replace=TRUE)],
          v7=c(TRUE,FALSE)[base::sample(1:2,n,replace=TRUE)]
      ))
  </code></pre>
  
  <p>The command that converts R df to Spark df was running for several hours. Had to cancel. Please assist.</p>
  
  <hr>
  
  <p><strong>EDIT 2016-12-14:</strong></p>
  
  <p>The above was attempted using Spark 1.6.1 and R 3.2.0. I have recently tried this using Spark 2.0.2 (latest) and R 3.2.5, and I ran into the same issue.</p>
  
  <p>Any help would be greatly appreciated.</p>