title: how to control potential fork bomb caused by mclapply, tried ulimit but didn't work
body: <p>I am using <code>mclapply</code> in my R script for parallel computing. It saves overall memory usage and it is fast so I want to keep it in my script. However, one thing I noticed is that the number of child processes generated during running the script is more than the number of cores I specified using <code>mc.cores</code>. Specifically, I am running my script on a server with 128 cores. And when I run my script, I set <code>mc.cores</code> to 18. During the running of the script, I checked the processes related to my script using <code>htop</code>. First, I can find 18 processes like this:
  <a href="https://i.stack.imgur.com/9vwxb.png" rel="nofollow noreferrer">enter image description here</a></p>
  <p>3_GA_optimization.R is my script. This all look good. But I also found more than 100 processes running at the same time with similar memory and CPU usage. The screenshot below shows some of them:
  <a href="https://i.stack.imgur.com/fky0A.png" rel="nofollow noreferrer">enter image description here</a></p>
  <p>The problem of this is that although I only required 18 cores, the script actually uses all the 128 cores on the server and this makes the server very slow. So my first question is why is this happening? And what is the difference between these processes with green color compared to the 18 processes with black color?</p>
  <p>My second question is that I tried to use <code>ulimit -Su 100</code> to set the soft limit of maximum number of processes that I can use before running <code>Rscript 3_GA_optimization.R</code>. I chose 100 based on the current number of processes I am using before running the script and the number of cores I want to use when running the script. However, I got an error saying:</p>
  <p>Error in mcfork():
  unable to fork, possible reason: Resource temporarily unavailable</p>
  <p>So it seems that <code>mclapply</code> has to generate a lot more processes than <code>mc.cores</code> in order for the script to run, which is confusing to me. So my second question is that why does <code>mclapply</code> behaves in this way? Is there any other way to fix the total number of cores <code>mclapply</code> can use?</p>