title: RegCM, MPICH, Computer Clustering
body: <p><strong>Background:</strong>
  I need to run a huge computation for climate simulation with over <strong><code>800 [GB]</code></strong> of data ( for the past 50 years and future 80 years ).</p>
  
  <p>For this, I'm using RegCM4 based in linux. I am using Ubuntu. The most powerful system we have has some Intel XEON processor with 20 cores. Also we have almost 20 smaller less powerful Intel i7 octa-core processors.</p>
  
  <p>To run the simulations, the single system will take more than a month.</p>
  
  <p>So, I've been trying to set up computer clusters with available resources.<br>
  (FYI: RegCM allows parallel processing with <code>mpi</code>.)</p>
  
  <p><strong>Specs::</strong></p>
  
  <pre><code>Computer socket cores_per_socket threads_per_core CPUs   RAM   Hard_drives 
  node0    1      10               2                20   32 GB   256 GB  + 2 TB-HDD
  node1    1       4               2                 8    8 GB             1 TB-HDD
  node2    1       4               2                 8    8 GB             1 TB-HDD
  </code></pre>
  
  <p>-> I use <code>mpich</code> v3 ( I don't remember exact version no. )</p>
  
  <p>And so on... ( all the nodes other than <code>node0</code> are the same as <code>node1</code>.)<br>
  All nodes have <strong><code>1 Gbps</code></strong> supported ethernet cards.<br>
  For test purpose I have set up a small simulation work for analyzing 6 days of climate. All test simulations used same parameters and model settings.</p>
  
  <p>All nodes boot from their own HDD.<br>
  <code>node0</code> runs on Ubuntu 16.04 LTS.<br>
  other nodes run Ubuntu 14.04 LTS.</p>
  
  <p><strong>How I started?</strong>
  I followed steps as in <a href="http://mpitutorial.com/tutorials/running-an-mpi-cluster-within-a-lan/" rel="nofollow noreferrer">here</a>.</p>
  
  <ol>
  <li>Connected <code>node1</code> and <code>node2</code> with a Cat 6 cable, assigned them static IP-s. (left <code>node0</code> for now) - edited <code>/etc/hosts</code> with IP-s and corresponding names - <code>node1</code> and <code>node2</code> as given in table above</li>
  <li>setup password-less login with ssh in both - success</li>
  <li>created a folder in <code>/home/user</code> in <code>node1</code> (which will be master in this test) and exported the folder ( <code>/etc/exports</code> ), mounted this folder over <strong><code>NFS</code></strong> to <code>node2</code> and edited <code>/etc/fstab</code> in <code>node2</code> - success</li>
  <li>Ran my <code>regcm</code> over the cluster using 14 cores of both machines - success</li>
  <li>I have used : <code>iotop</code>, <code>bmon</code>, <code>htop</code> to monitor disk read/write, network traffic and CPU usage respectively.</li>
  </ol>
  
  <blockquote>
    <p><strong><code>$ mpirun -np 14 -hosts node0,node1 ./bin/regcmMPI test.in</code></strong></p>
  </blockquote>
  
  <p><strong>Result of this test</strong><br>
  Faster computation over a single node processing</p>
  
  <hr>
  
  <p><strong>Now I tried the same with <code>node0</code></strong>  (see above for computer specs)</p>
  
  <p>-> I am working on SSD in <code>node0</code>.<br>
  -> works fine but the problem is time factor when connected in cluster.</p>
  
  <p>Here's the summary of results::
  - first using <code>node0</code> only - no use of cluster</p>
  
  <blockquote>
    <p><strong><code>$ mpirun -np 20 ./bin/regcmMPI test.in</code></strong></p>
  </blockquote>
  
  <pre><code>nodes   no.of_cores_used    run_time_reported_by_regcm_in_sec   actual time taken in sec (approx)
  node0   20                  59.58                                60
  node0   16                  65.35                                66
  node0   14                  73.33                                74
  </code></pre>
  
  <p>this is okay</p>
  
  <p>Now, using cluster<br>
  <sub>( use following ref to understand the table below )</sub>:</p>
  
  <blockquote>
    <p><strong><code>rt</code></strong> = <em>CPU run time reported by regcm in sec</em></p>
    
    <p><strong><code>a-rt</code></strong> = <em>actual time taken in sec (approx)</em></p>
    
    <p><strong><code>LAN</code></strong> = <em>Max LAN speed achieved (Receive/Send) in MBps</em></p>
    
    <p><strong><code>disk(0 / 1)</code></strong> = <em>Max Disk write speed at <code>node0</code> / at <code>node1</code>  in MBps</em></p>
  </blockquote>
  
  <pre><code>nodes*  cores   rt      a-rt    LAN     disk(  0 /  1 )
  1,0    16       148     176     100/30        90 / 50
  0,1    16       145     146      30/30         6 /  0
  1,0    14       116     143     100/25        85 / 75
  0,1    14       121     121      20/20         7 /  0
  </code></pre>
  
  <blockquote>
    <p>*note:</p>
    
    <p>1,0 (eg. for 16 cores) means: <strong><code>$ mpirun -np 16 -hosts node1,node0  ./bin/regcmMPI test.in</code></strong></p>
    
    <p>0,1 (eg. for 16 cores) means: <strong><code>$ mpirun -np 16 -hosts node0,node1 ./bin/regcmMPI test.in</code></strong></p>
  </blockquote>
  
  <p><em>Actual run time was calculated manually using start and end time reported by regcm.</em></p>
  
  <p>We can see above that LAN-usage and drive write speed was significantly different for two options - 1. passing <strong><code>node1,node0</code></strong> as host ; and 2. passing <strong><code>node0,node1</code></strong> as host ---- note the order.</p>
  
  <p>Also time for running in single node is faster than running in cluster. <strong>Why ?</strong></p>
  
  <p>I also ran another set of test, this time using hostfile (named hostlist) whose content were:</p>
  
  <pre><code>node0:16
  node1:6
  </code></pre>
  
  <p>Now I ran the following script</p>
  
  <blockquote>
    <p><strong><code>$ mpirun -np 22 -f hostlist ./bin/regcmMPI test.in</code></strong></p>
  </blockquote>
  
  <p>CPU run time was reported <strong><code>101 [s]</code></strong>, actual run time was <strong><code>1 min 42 sec</code></strong> ( <code>102 [s]</code> ), LAN speed achieved was around <strong><code>10-15 [MB/s]</code></strong>, disk write speed was around <strong><code>7 [MB/s]</code></strong>.</p>
  
  <p>The best result was obtained when I used the same hostfile setting and ran code with 20 processors thus under-subscribing</p>
  
  <blockquote>
    <p><strong><code>$ mpirun -np 20 -f hostlist ./bin/regcmMPI test.in</code></strong></p>
  </blockquote>
  
  <pre><code>CPU runtime     : 90 [s]
  Actual run time : 91 [s]
  LAN             : 10 [MB/s]
  </code></pre>
  
  <p>When I changed cores from 20 downto 18, run time increased to <strong><code>102 [s]</code></strong>.</p>
  
  <p>I have <strong>not</strong> yet connected <code>node2</code> to the system.</p>
  
  <hr>
  
  <h2><strong>Questions:</strong></h2>
  
  <ol>
  <li>Is there a way to achieve faster speed in computation ? Am I doing something wrong ?</li>
  <li>The computation time for single machine with 14 cores is faster than cluster with 22 cores or 20 cores. Why is it happening ?</li>
  <li>What is the optimum number of cores that can be used to achieve time efficiency ?</li>
  <li>How can I achieve best performance with available resources?</li>
  <li>Are there any best mpich usage manual that can answer my questions? (I could not find any such info)</li>
  <li>Sometimes using fewer cores give faster completion time than using higher cores even though I am not using all available cores leaving 1 or 2 cores for OS and other operations in individual nodes. Why is it happening?</li>
  </ol>