<p>According to the doc in <a href="https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/mpi" rel="nofollow noreferrer">Tensorflow git repo</a>，actually tf utilizes <a href="https://github.com/grpc/grpc" rel="nofollow noreferrer">gRPC</a> library by detault, which is based on HTTP2 protocol, rather than TCP/IP protocol, and <a href="https://arxiv.org/pdf/1603.02339.pdf" rel="nofollow noreferrer">this paper</a> should give you some insight, hope this information is useful.</p>
Implications of using MPI with TensorFlow
<p>I come from a sort of HPC background and I am just starting to learn about machine learning in general and TensorFlow in particular.  I was initially surprised to find out that distributed TensorFlow is designed to communicate with TCP/IP by default though it makes sense in hindsight given what Google is and the kind of hardware it uses most commonly.</p>

<p>I am interested in experimenting with TensorFlow in a parallel way with MPI on a cluster.  From my perspective, this should be advantageous because latency should be much lower due to MPI's use of Remote Direct Memory Access (RDMA) across machines without shared memory.</p>

<p>So my question is, why doesn't this approach seem to be more common given the increasing popularity of TensorFlow and machine learning ?  Isn't latency a bottleneck ?  Is there some typical problem that is solved, that makes this sort of solution impractical?  Are there likely to be any meaningful differences between calling TensorFlow functions in a parallel way vs implementing MPI calls inside of the TensorFlow library ?</p>

<p>Thanks</p>