title: Parallel programming in R: Error writing to connection
body: <p>I am using the <code>foreach</code> package to run my code in parallel. Although it works correctly for "small" data sets, it fails to run for larger data sets. More specifically, it throws this error:</p>
  
  <blockquote>
    <p>Error in serialize(data, node$con) : error writing to connection</p>
  </blockquote>
  
  <p>This is my code:</p>
  
  <pre><code>library(foreach)
  library(doParallel)
  library(iterators)
  
  numCores = detectCores() - 1
  clm = makePSOCKcluster(numCores)
  registerDoParallel(clm)
  
  results = foreach(i = 1:1000) %dopar% {
      myFunction(i, otherArguments)
  }
  stopCluster(clm)
  </code></pre>
  
  <p>While running, I notice in the Task Manager that the Memory almost reaches 100% before the program stops. I tried using various numbers of cores and tried setting the <code>memory.limit()</code>, but not in a systematic way and the problem persisted. What could my problem be and how can I resolve it?</p>
  
  <p>Computer: Windows 10 64bit, 28GB Ram, CPU 8 cores, R3.2.2 (64 bit)</p>
  
  <p>I know something similar was posted <a href="https://stackoverflow.com/questions/28503208/doparallel-error-in-r-error-in-serializedata-nodecon-error-writing-to-con">here</a>, but the issue was not resolved then.</p>