title: Why is the performance of Apache Spark worse than R on a local cluster?
body: <p>I am comparing the performance of R and Apache Spark on a local machine and R seems to be doing much better. Is that because I am not using a cluster or am I doing something wrong?</p>
  
  <p><strong>Create data (create_data.R):</strong></p>
  
  <pre><code>options = commandArgs(trailingOnly = TRUE)
  rows = as.numeric(options[1])
  
  perday = 365 / (rows-1) * 6
  dates = seq(as.Date('2010-01-01'), as.Date('2015-12-31'), by=perday)
  rows = length(dates)
  ids = sample(paste0("ID", seq(1:10000)), rows, replace=TRUE)
  sales = rpois(rows,50)
  categories = sample(paste("Category", sprintf("%02d",seq(1:10))), rows, replace=TRUE)
  data = data.frame(dates, ids, sales, categories)
  
  write.csv(data, "/home/phil/performance/data.csv", row.names=FALSE)
  </code></pre>
  
  <p><strong>Test R (cut.R):</strong></p>
  
  <pre><code>suppressMessages(suppressWarnings(require(dplyr, quietly=TRUE)))
  
  data = read.csv("data.csv")
  first_purchase = head(data[order(data$dates, data$ids),],1)
  
  print(first_purchase)
  </code></pre>
  
  <p><strong>Test Spark (cut.py):</strong></p>
  
  <pre><code>from pyspark import SparkContext
  
  sc = SparkContext("local")
  
  rdd = sc.textFile("data.csv", 2)
  
  # Get rid of header
  header = rdd.take(1)[0]
  rdd = rdd.filter(lambda line: line != header)
  
  rdd = rdd.map(lambda line: line.split(","))
  first_purchase = rdd.takeOrdered(1, lambda x: [x[0],x[1]])[0]
  print(first_purchase)
  </code></pre>
  
  <p><strong>Run complete test (run_tests.sh):</strong></p>
  
  <pre><code>echo "Creating data"
  Rscript create_data.R 5000000
  wc -l data.csv
  
  echo "Testing R"
  time Rscript cut.R
  
  echo "Testing Spark"
  time spark-submit cut.py
  </code></pre>
  
  <p><strong>Output of the tests:</strong></p>
  
  <pre><code>$ . run_test.sh
  Creating data
  5000001 data.csv
  Testing R
            dates  ids sales  categories
  1264 2010-01-01 ID10    60 Category 01
  
  real    0m12.689s
  user    0m12.498s
  sys     0m0.187s
  Testing Spark
  [u'2010-01-01', u'"ID10"', u'60', u'"Category 01"']
  
  real    0m17.029s
  user    0m7.388s
  sys     0m0.392s
  </code></pre>
  
  <p>I am running this on a Ubuntu in a VirtualBox with Windows 7 as host system, if that makes a difference.</p>