title: JVM crashes when attempting to run multiple invocations on a high-memory, many CPU server
body: <p>I've written some Java-based, single-threaded, genomics software that runs on one sample at a time, and I have hundreds of samples to process.  I have access to a machine with 64 CPUs and a terabyte of RAM, and there are no other users on the system.  The maximum heap size requested for each invocation is 8 gigabytes.  I expect I should be able to invoke 30 instances of my code simultaneously (assuming two threads - my main thread and a GC thread?).  I'm only attempting to dispatch 20 at a time (using a makefile and the -j20 argument).  In practice, however, only 5 run.  The rest fail with the message:</p>
  
  <pre><code># There is insufficient memory for the Java Runtime Environment to continue.
  # Cannot create GC thread. Out of system resources.
  </code></pre>
  
  <p>Suspicious that this had more to do with the simultaneous invocations than actual resource limitations, I implemented a random several-second delay in my execution loop for each invocation of my program.  That gets me up to 10 programs running simultaneously, rather than 5, with the same failure message.</p>
  
  <p>Questions:</p>
  
  <ol>
  <li><p>Why does attempting to invoke dozens of instances of the JVM simultaneously fail this way, despite doing so on a system that clearly has the resources available?</p></li>
  <li><p>Why does my hack implementing the dispatch delay clear up some of the problem?</p></li>
  <li><p>What's a better way to get all 20 instances running simultaneously?</p></li>
  </ol>