title: parallel processing in R using snow
body: <p>I have 1000's of list and each list has multiple time series. I would like to apply forecasting to each element in the list. This has became an intractable problem interms of computing resources. I don't have backgrounder in parallel computing or advanced R programming. Any help would be greatly appreciated.</p>
  
  <p>I have created dummy list. Basically, dat.list is similar to what I'm working on. </p>
  
  <pre><code>library("snow")
  library("plyr")
  library("forecast")
  
      ## Create Dummy Data
  
      z &lt;- ts(matrix(rnorm(30,10,10), 100, 3), start = c(1961, 1), frequency = 12)
      lam &lt;- 0.8
      ap &lt;- list(z=z,lam=lam)
  
  ## forecast using lapply
  
      z &lt;- ts(matrix(rnorm(30,10,10), 100, 3), start = c(1971, 1), frequency = 12)
      lam &lt;- 0.5
      zp &lt;- list(z=z,lam=lam)
  
      dat.list &lt;- list(ap=ap,zp=zp)
  
      xa &lt;- proc.time()
      tt &lt;- lapply(dat.list,function(x) lapply(x$z,function(y) (forecast::ets(y))))
      xb &lt;- proc.time()
  </code></pre>
  
  <p>The above code gives me what I need. I would like apply parrallel processing to both lapply in the code above.
  So I have attempted to use snow package and an example shown in this <a href="http://www.r-bloggers.com/quick-guide-to-parallel-r-with-snow/" rel="nofollow">site</a>.  </p>
  
  <pre><code>  ## Parallel Processing
  
  
      clus &lt;- makeCluster(3)
      custom.function &lt;- function(x) lapply(x$z,function(y) (forecast::ets(y)))
      clusterExport(clus,"custom.function")
  
      x1 &lt;- proc.time()
      tm &lt;- parLapply(clus,dat.list,custom.function)
      x2&lt;-proc.time()
  
      stopCluster(clus)
  </code></pre>
  
  <p>Below are my questions, </p>
  
  <ol>
  <li>For some reason, the output of tm is differenct for the non parallel
  version. the forecast function ets is applied to every single data point as opposed to the element in the list.</li>
  </ol>
  
  <p>Non parallel:</p>
  
  <pre><code>summary(tt)
     Length Class  Mode
  ap 3      -none- list
  zp 3      -none- list
  </code></pre>
  
  <p>Parallel Version:</p>
  
  <pre><code>    summary(tm)
         Length Class  Mode
      ap 300    -none- list
      zp 300    -none- list
  </code></pre>
  
  <ol start="2">
  <li><p>My second question is how should I parallelize  the lapply in the
  custom function, basically a nested parLapply</p>
  
  <p>custom.function &lt;- function(x) parLapply(clus,x$z,function(y) (forecast::ets(y))) ## Not working</p></li>
  </ol>
  
  <p>Many Thanks for your help</p>