title: OpenMPI cannot fully utilize 10 GE
body: <p>I tried to perform data exchange between two machines connected with 10GE. The size of data is large enough (8 GB) to expect network utilization near the maximum. But surprisingly I observed absolutely different behavior.  </p>
  
  <p>To check the throughput I have used two different programs - <code>nethogs</code> and <code>nload</code>, both of them show that network utilization is much lower than expected. Moreover the results are unpredictable - sometimes in and out channels are utilized simultaneously, but sometimes transmission and reception are separated as if there is a half-duplex channel. Sample output of <code>nload</code>:</p>
  
  <pre class="lang-none prettyprint-override"><code>Device enp1s0f0 [192.168.0.11] (1/1):
  ======================================================================================================================
  Incoming:
  
  
  
  
                                                        |||||||||||||||||||
                                                       .###################
                                                       ####################|
                                                       #####################      Curr: 0.00 GBit/s
                                                       #####################      Avg: 2.08 GBit/s
                                                      .#####################      Min: 0.00 GBit/s
                                                      #######################     Max: 6.32 GBit/s
                                                      #######################     Ttl: 57535.38 GByte
  Outgoing:
  
  
  
  
                                    ||||||||||||||||||
                                    ##################
                                   |##################
                                   ###################|
                                   ####################                           Curr: 0.00 GBit/s
                                   ####################                           Avg: 2.09 GBit/s
                                  .####################                           Min: 0.00 GBit/s
                                  #####################.                          Max: 6.74 GBit/s
                                  ######################                          Ttl: 57934.64 GByte
  </code></pre>
  
  <p>The code I use is here:</p>
  
  <pre class="lang-cpp prettyprint-override"><code>int main(int argc, char** argv) {
      boost::mpi::environment env{};
      boost::mpi::communicator world{};
      boost::mpi::request reqs[2];
  
      int k = 10;
      if(argc &gt; 1)
          k = std::atoi(argv[1]);
  
      uint64_t n = (1ul &lt;&lt; k);
  
      std::vector&lt;std::complex&lt;double&gt;&gt; sv(n, world.rank());
      std::vector&lt;std::complex&lt;double&gt;&gt; rv(n);
  
      int dest = world.rank() == 0 ? 1 : 0;
      int src = dest;
  
      world.barrier();
      reqs[0] = world.irecv(src, 0, rv.data(), n);
      reqs[1] = world.isend(dest, 0, sv.data(), n);
      boost::mpi::wait_all(reqs, reqs + 2);
  
      return 0;
  }
  </code></pre>
  
  <p>And here is the command I use to run on cluster:</p>
  
  <pre><code>mpirun --mca btl_tcp_if_include 192.168.0.0/24 --hostfile ./host_file -n 2 --bind-to core /path/to/shared/folder/mpi_exp 29
  </code></pre>
  
  <p><code>29</code> here means that 2^(29 + 4) = 8 GBytes will be sent</p>
  
  <p>What I have done:</p>
  
  <ul>
  <li><p>Proved that there is no hardware problem by successful saturation of the channel with <code>netcat</code>.</p></li>
  <li><p>Checked with <code>tcpdump</code> that the size of TCP packets during the communication is unstable and rarely reach the maximum size (in <code>netcat</code> case it is stable).</p></li>
  <li><p>Checked with <code>strace</code> that socket operations are correct.</p></li>
  <li><p>Checked TCP parameters in <code>sysctl</code> - they are ok.</p></li>
  </ul>
  
  <p>Could you please advise me why OpenMPI doesn't work as expected? </p>
  
  <p><strong>EDIT (14.08.2018):</strong></p>
  
  <p>Finally I was able to continue to dig into this problem. Below is the output of OSU bandwidth benchmark (it was run without any mca options):</p>
  
  <pre class="lang-none prettyprint-override"><code># OSU MPI Bandwidth Test v5.3
  # Size      Bandwidth (MB/s)
  1                       0.50
  2                       0.98
  4                       1.91
  8                       3.82
  16                      6.92
  32                     10.32
  64                     22.03
  128                    43.95
  256                    94.74
  512                   163.96
  1024                  264.90
  2048                  400.01
  4096                  533.47
  8192                  640.02
  16384                 705.02
  32768                 632.03
  65536                 667.29
  131072                842.00
  262144                743.82
  524288                654.09
  1048576               775.50
  2097152               759.44
  4194304               774.81
  </code></pre>
  
  <p>Actually I think that such poor performance is caused by CPU bound. Each MPI  process is single-threaded by default, and it is just not able to saturate 10GE channel.</p>
  
  <p>I know it is possible to communicate with several threads by enabling multithreading when building OpenMPI. But such approach will lead to increased complexity on application level. </p>
  
  <p>So is it possible to have multithreaded sending/receiving in OpenMPI internally on the level responsible for point-to-point data transfer?</p>