<p>Another possibility is to use a distributed version of TensorFlow, which automatically handles the data distribution and execution on multiple nodes by using MPI in the backend.</p>

<p>We have recently developed one such version at MaTEx: <a href="https://github.com/matex-org/matex" rel="nofollow noreferrer">https://github.com/matex-org/matex</a>, and a paper describing <a href="https://arxiv.org/abs/1704.04560" rel="nofollow noreferrer">https://arxiv.org/abs/1704.04560</a></p>

<p>It does synchronous training and provides several parallel dataset reader format.</p>

<p>We will be happy to help you if you need more help!</p>
What is the reason to use parameter server in distributed tensorflow learning?
<p><strong>Short version:</strong> can't we store variables in one of the workers and not use parameter servers?</p>

<p><strong>Long version:</strong>
I want to implement synchronous distributed learning of neural network in tensorflow. I want each worker to have a full copy of the model during training.</p>

<p>I've read <a href="https://www.tensorflow.org/versions/r0.10/how_tos/distributed/index.html%20distributed%20tensorflow%20tutorial" rel="noreferrer">distributed tensorflow tutorial</a> and <a href="https://github.com/tensorflow/models/tree/master/inception" rel="noreferrer">code of distributed training imagenet</a> and didn't get why do we need parameter servers.</p>

<p>I see that they are used for storing values of variables and replica_device_setter takes care that variables are evenly distributed between parameter servers (probably it does something more, I wasn't able to fully understand the code).</p>

<p>The question is: why don't we use one of the workers to store variables? Will I achieve that if I use  </p>

<pre><code>with tf.device('/job:worker/task:0/cpu:0'):
</code></pre>

<p>instead of</p>

<pre><code>with tf.device(tf.train.replica_device_setter(cluster=cluster_spec)):
</code></pre>

<p>for Variaibles?
If that works is there downside comparing to solution with parameter servers?</p>