title: Determining ideal number of cores in MPI
body: <p>I'm trying to determine the ideal number of cores for my model (in this example I'm using xbeach but I'm having the same issue when running other models such as SWAN) by doing a speedup test, i.e. I'm running the same code <code>mpiexec -n &lt;np&gt; mymodel.exe</code> with <code>&lt;np&gt; = 2, 4, 8, 12, 16, 18, 20, 24, 32, and 36</code> cores/processes and see how long it takes. I ran the test multiple times, each with different results - see image</p>
  
  <p><a href="https://i.stack.imgur.com/gzsq6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gzsq6.png" alt="Computing time with different number of cores"></a></p>
  
  <p>To my understanding MPI tries to distribute all tasks evenly across all processors (so having other applications running in the background might slow things down) but I'm still confused why I get SUCH a large range of computing times for the same code with the same number of cores/tasks. Why could it be that my model takes 18 minutes to run on 18 cores one time and then 128 minutes the next time I run it?</p>
  
  <p>I'm using MPICH2 1.4.1p1 on Windows 10 on a machine with 2 NUMA nodes with 36 cores each so any potential background tasks should easily be handled on the remaining cores/node. It also doesn't look like I'm running out of memory.</p>
  
  <p>To summarize my questions are:</p>
  
  <ol>
  <li>Is it normal to have such a large range in computation time when using MPI?</li>
  <li>If it is - why?</li>
  <li>Is there a better way than mine to determine the ideal number of cores for an MPI program?</li>
  </ol>