title: Why is the parallel package slower than just using apply?
body: <p>I am trying to determine when to use the <code>parallel</code> package to speed up the time necessary to run some analysis. One of the things I need to do is create matrices comparing variables in two data frames with differing number of rows. I asked a question as to an efficient way of doing on <a href="https://stackoverflow.com/questions/14485040/is-there-an-efficient-way-of-comparing-two-data-frames">StackOverflow</a> and wrote about tests on my <a href="https://web.archive.org/web/20171209073305/http://jason.bryer.org:80/posts/2013-01-24/Comparing_Two_Data_Frames.html" rel="noreferrer">blog</a>. Since I am comfortable with the best approach I wanted to speed up the process by running it in parallel. The results below are based upon a 2ghz i7 Mac with 8gb of RAM. I am surprised that the <code>parallel</code> package, the <code>parSapply</code> funciton in particular, is worse than just using the <code>apply</code> function. The code to replicate this is below. Note that I am currently only using one of the two columns I create but eventually want to use both.</p>
  
  <p><a href="https://i.stack.imgur.com/xhz6n.png" rel="noreferrer"><img src="https://i.stack.imgur.com/xhz6n.png" alt="Execution Time"></a><br>
  <sub>(source: <a href="http://jason.bryer.org/images/ParalleVsApplyTiming.png" rel="noreferrer">bryer.org</a>)</sub>  </p>
  
  <pre><code>require(parallel)
  require(ggplot2)
  require(reshape2)
  set.seed(2112)
  results &lt;- list()
  sizes &lt;- seq(1000, 30000, by=5000)
  pb &lt;- txtProgressBar(min=0, max=length(sizes), style=3)
  for(cnt in 1:length(sizes)) {
      i &lt;- sizes[cnt]
      df1 &lt;- data.frame(row.names=1:i, 
                        var1=sample(c(TRUE,FALSE), i, replace=TRUE), 
                        var2=sample(1:10, i, replace=TRUE) )
      df2 &lt;- data.frame(row.names=(i + 1):(i + i), 
                        var1=sample(c(TRUE,FALSE), i, replace=TRUE),
                        var2=sample(1:10, i, replace=TRUE))
      tm1 &lt;- system.time({
          df6 &lt;- sapply(df2$var1, FUN=function(x) { x == df1$var1 })
          dimnames(df6) &lt;- list(row.names(df1), row.names(df2))
      })
      rm(df6)
      tm2 &lt;- system.time({
          cl &lt;- makeCluster(getOption('cl.cores', detectCores()))
          tm3 &lt;- system.time({
              df7 &lt;- parSapply(cl, df1$var1, FUN=function(x, df2) { x == df2$var1 }, df2=df2)
              dimnames(df7) &lt;- list(row.names(df1), row.names(df2))
          })
          stopCluster(cl)
      })
      rm(df7)
      results[[cnt]] &lt;- c(apply=tm1, parallel.total=tm2, parallel.exec=tm3)
      setTxtProgressBar(pb, cnt)
  }
  
  toplot &lt;- as.data.frame(results)[,c('apply.user.self','parallel.total.user.self',
                            'parallel.exec.user.self')]
  toplot$size &lt;- sizes
  toplot &lt;- melt(toplot, id='size')
  
  ggplot(toplot, aes(x=size, y=value, colour=variable)) + geom_line() + 
      xlab('Vector Size') + ylab('Time (seconds)')
  </code></pre>