<p>I've also encountered this issue.
See <a href="https://arxiv.org/abs/1706.02677" rel="nofollow noreferrer">Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour</a> from Facebook which addresses the same issue. The suggested solution is simply to scale up the learning rate by k (after some reasonable warm-up epochs) for k GPUs.</p>

<p>In practice I've found out that simply summing up the gradients from the GPUs (rather than averaging them) and using the original learning rate sometimes does the job as well.</p>
Multi GPU architecture, gradient averaging - less accurate model?
<p>When I execute the cifar10 model as described at <a href="https://www.tensorflow.org/tutorials/deep_cnn" rel="noreferrer">https://www.tensorflow.org/tutorials/deep_cnn</a> I achieve 86% accuracy after approx 4 hours using a single GPU , when I utilize 2 GPU's the accuracy drops to 84% but reaching 84% accuracy is faster on 2 GPU's than 1. </p>

<p>My intuition is 
that average_gradients function as defined at <a href="https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py" rel="noreferrer">https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py</a> returns a less accurate gradient value as an average of gradients will be less accurate than the actual gradient value. </p>

<p>If the gradients are less accurate then the parameters than control the function that is learned as part of training is less accurate. Looking at the code (<a href="https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py" rel="noreferrer">https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py</a>) why is averaging the gradients over multiple GPU's less accurate than computing the gradient on a single GPU ?</p>

<p>Is my intuition of averaging the gradients producing a less accurate value correct ?</p>

<p>Randomness in the model is described as : </p>

<pre><code>The images are processed as follows:
They are cropped to 24 x 24 pixels, centrally for evaluation or randomly for training.
They are approximately whitened to make the model insensitive to dynamic range.
For training, we additionally apply a series of random distortions to artificially increase the data set size:

Randomly flip the image from left to right.
Randomly distort the image brightness.
Randomly distort the image contrast.
</code></pre>

<p>src : <a href="https://www.tensorflow.org/tutorials/deep_cnn" rel="noreferrer">https://www.tensorflow.org/tutorials/deep_cnn</a></p>

<p>Does this have an effect on training accuracy ?</p>

<p>Update : </p>

<p>Attempting to investigate this further, the loss function value training with different number of GPU's.</p>

<pre><code>Training with 1 GPU : loss value : .7 , Accuracy : 86%
Training with 2 GPU's : loss value : .5 , Accuracy : 84%
</code></pre>

<p>Shouldn't the loss value be lower for higher for higher accuracy, not vice versa ?</p>