title: Error in unserialize(socklist[[n]]) : error reading from connection
body: <p>I'm using R and I'm running a parallel code on 7 cores on a Linux machine.</p>
  
  <p>When using a small dataset, my code takes about 2 hours and works fine.
  When using a 6x larger dataset, the codes takes much longer (probably because it needs to swap), but then will randomly end, sometimes at 10%, sometimes at 18, 20, 30% or so. Looks totally random. RAM usage is usually at about 90%, SWAP usage at &lt;50%.</p>
  
  <p>I'm using <code>foreach</code> with the <code>doSNOW</code> backend.
  This is the error code:</p>
  
  <pre><code>Error in unserialize(socklist[[n]]) : error reading from connection
  Calls: %dopar% ... tryCatch -&gt; tryCatchList -&gt; tryCatchOne -&gt; &lt;Anonymous&gt;
  Execution halted
  
   *** caught bus error ***
  address 0x7f829d2adbd0, cause 'non-existent physical address'
  An irrecoverable exception occurred. R is aborting now ...
  </code></pre>
  
  <p>And this is the SNOW outfile obtained by setting <code>outfile="outfile.out"</code> in the <code>makeCluster</code> call:</p>
  
  <pre><code>starting worker for localhost:11567 
  Type: EXEC 
  Loading required package: MASS
  Loading required package: survival
  Loading required package: sp
  
  Attaching package: 'raster'
  
  The following objects are masked from 'package:MASS':
  
      area, select
  
  
  Attaching package: 'data.table'
  
  The following object is masked from 'package:raster':
  
      shift
  
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  Type: EXEC 
  
   *** caught bus error ***
  address 0x7ffada863636, cause 'non-existent physical address'
  
  Traceback:
   1: .Call("Rsx_nc4_get_vara_double", as.integer(ncid), as.integer(varid),     as.integer(c.start), as.integer(c.count), fixmiss, imvstate,     as.double(passed_missval), PACKAGE = "ncdf4")
   2: ncvar_get_inner(ncid2use, varid2use, nc$var[[li]]$missval, addOffset,     scaleFact, start = start, count = count, verbose = verbose,     signedbyte = signedbyte, collapse_degen = collapse_degen)
   3: getfun(nc, varid = zvar, start = start, count = count)
   4: .readBrickCellsNetCDF(x, cells, layer, nl)
   5: .cellValues(x, i)
   6: .doExtract(x, i, drop = drop)
   7: por[i]
   8: por[i]
   9: as.vector(por[i])
  10: mainF(as.vector(por[i]))
  11: eval(expr, envir, enclos)
  12: eval(.doSnowGlobals$expr, envir = .doSnowGlobals$exportenv)
  13: doTryCatch(return(expr), name, parentenv, handler)
  14: tryCatchOne(expr, names, parentenv, handlers[[1L]])
  15: tryCatchList(expr, classes, parentenv, handlers)
  16: tryCatch(eval(.doSnowGlobals$expr, envir = .doSnowGlobals$exportenv),     error = function(e) e)
  17: fun(quote(list(i = 23339L)))
  18: do.call("fun", lapply(args, enquote))
  19: docall(msg$data$fun, msg$data$args)
  20: doTryCatch(return(expr), name, parentenv, handler)
  21: tryCatchOne(expr, names, parentenv, handlers[[1L]])
  22: tryCatchList(expr, classes, parentenv, handlers)
  23: tryCatch(docall(msg$data$fun, msg$data$args), error = handler)
  24: doTryCatch(return(expr), name, parentenv, handler)
  25: tryCatchOne(expr, names, parentenv, handlers[[1L]])
  26: tryCatchList(expr, classes, parentenv, handlers)
  27: tryCatch({    msg &lt;- recvData(master)    cat(paste("Type:", msg$type, "\n"))    if (msg$type == "DONE") {        closeNode(master)        break    }    else if (msg$type == "EXEC") {        success &lt;- TRUE        handler &lt;- function(e) {            success &lt;&lt;- FALSE            structure(conditionMessage(e), class = c("snow-try-error",                 "try-error"))        }        t1 &lt;- proc.time()        value &lt;- tryCatch(docall(msg$data$fun, msg$data$args),             error = handler)        t2 &lt;- proc.time()        value &lt;- list(type = "VALUE", value = value, success = success,             time = t2 - t1, tag = msg$data$tag)        sendData(master, value)    }}, interrupt = function(e) NULL)
  28: slaveLoop(makeSOCKmaster(master, port))
  29: eval(expr, envir, enclos)
  30: eval(quote({    master &lt;- "localhost"    port &lt;- ""    snowlib &lt;- Sys.getenv("R_SNOW_LIB")    outfile &lt;- Sys.getenv("R_SNOW_OUTFILE")    args &lt;- commandArgs()    pos &lt;- match("--args", args)    args &lt;- args[-(1:pos)]    for (a in args) {        pos &lt;- regexpr("=", a)        name &lt;- substr(a, 1, pos - 1)        value &lt;- substr(a, pos + 1, nchar(a))        switch(name, MASTER = master &lt;- value, PORT = port &lt;- value,             SNOWLIB = snowlib &lt;- value, OUT = outfile &lt;- value)    }    if (!(snowlib %in% .libPaths())) .libPaths(c(snowlib, .libPaths()))    library(methods)    library(snow)    if (port == "") port &lt;- getClusterOption("port")    sinkWorkerOutput(outfile)    cat("starting worker for", paste(master, port, sep = ":"),         "\n")    slaveLoop(makeSOCKmaster(master, port))}), new.env())
  31: eval(expr, envir, enclos)
  32: eval(expr, p)
  33: eval.parent(substitute(eval(quote(expr), envir)))
  34: local({    master &lt;- "localhost"    port &lt;- ""    snowlib &lt;- Sys.getenv("R_SNOW_LIB")    outfile &lt;- Sys.getenv("R_SNOW_OUTFILE")    args &lt;- commandArgs()    pos &lt;- match("--args", args)    args &lt;- args[-(1:pos)]    for (a in args) {        pos &lt;- regexpr("=", a)        name &lt;- substr(a, 1, pos - 1)        value &lt;- substr(a, pos + 1, nchar(a))        switch(name, MASTER = master &lt;- value, PORT = port &lt;- value,             SNOWLIB = snowlib &lt;- value, OUT = outfile &lt;- value)    }    if (!(snowlib %in% .libPaths()))         .libPaths(c(snowlib, .libPaths()))    library(methods)    library(snow)    if (port == "")         port &lt;- getClusterOption("port")    sinkWorkerOutput(outfile)    cat("starting worker for", paste(master, port, sep = ":"),         "\n")    slaveLoop(makeSOCKmaster(master, port))})
  An irrecoverable exception occurred. R is aborting now ...
  </code></pre>
  
  <p>The code is using 7 threads out of 8 of a local machine. The foreach call is made like so:</p>
  
  <pre><code>#Packages
  packageVec &lt;- c("RcppRoll", "FAdist", "fitdistrplus", "minpack.lm", "raster", "foreach", "data.table")
  #Register cluster
  cl &lt;- makeCluster(nThreads, outfile=paste0(dischargefile, ".out"))
  registerDoSNOW(cl)
  #Create progress bar
  pb &lt;- txtProgressBar(max = ncells, style = 3)
  progress &lt;- function(n) setTxtProgressBar(pb, n)
  SNOWopts &lt;- list(progress = progress)
  #Compute
  a &lt;-
      foreach(i=cells, .packages=packageVec, .combine='rbind', .options.snow = SNOWopts) %dopar% {
        mainF(as.vector(por[i]))
      }
  stopCluster(cl)
  </code></pre>
  
  <h1>mainF() function description</h1>
  
  <p>It's hard to produce a MRE for this function, as it is complicated. The code is <a href="http://pastebin.com/bip1UXvn" rel="nofollow">here</a>, and I'll describe what the function does below. I want to stress the fact that all of this works wonderfully on a small dataset, but fails on a large dataset, even if it contains the same data of the small dataset replicated several times.</p>
  
  <p><code>mainF()</code> is a function whose input, <code>por[i]</code>, is a 125000 element vector and whose output a 244 element vector. This function basically performs 240 rolling means on the input vector (using <code>RcppRoll</code>) and takes the maxima (using <code>data.table</code>) for each time slice (usually 15) for each mean, fits the data to a given distribution (using <code>FAdist</code>, <code>raster</code> and <code>fitdistrplus</code>), performs another loop for calculating another 240 values and fits (using <code>minpack.lm</code>) those values to a function to obtain more parameters. The 240 elements returned and the 4 fit parameters (2 for each fits) are returned.</p>
  
  <p>How can I fix the above error? What does it mean?</p>