title: Parallel processing in R - why more cores takes more time?
body: <p>I did a test where I parallelized 1000 jobs to N cores using mclapply in R (code below). I got some strange result, where up to a point (~40 cores), more cores speed up the process, but from that point it actually takes more time with more cores. What am I missing here?</p>
  <p>Thanks,
  Nitzan</p>
  <p><a href="https://i.stack.imgur.com/K5C87.png" rel="nofollow noreferrer">Number of cores (x-axis) as a function of time (y-axis)</a></p>
  <p>Here is the code I used where 'myfunction' is some reinforcement learning model</p>
  <pre><code>for (i in 1:12){
  mycores[i]=96-8*(i-1)
  print(mycores)
  start_time &lt;- Sys.time()
  
  df&lt;-
  lapply(1:3,     function(X){
  lapply(1:10,    function(Y){
  mclapply(1:1000,function(Z){
  myfunction(X,Y,Z),
  mc.cores=mycores[i])})})
  
  mytime[i]= Sys.time() - start_time
  }
  </code></pre>