<p>There is a decent discussion of this <a href="http://sebastianruder.com/optimizing-gradient-descent/index.html#parallelizinganddistributingsgd" rel="noreferrer">here</a> (not my content). Basically when you distribute SGD, you have to communicate gradients back and forth somehow between workers. This is inherently imperfect, and so your distributed SGD typically diverges from a sequential, single-worker SGD at least to some degree. It is also typically faster, so there is a trade off.</p>

<p><a href="https://arxiv.org/pdf/1412.6651.pdf" rel="noreferrer">[Zhang <em>et. al.</em>, 2015]</a> proposes one method for distributed SGD called elastic-averaged SGD. The paper goes through a stability analysis characterizing the behavior of the gradients under different communication constraints. It gets a little heavy, but it might shed some light on why you see this behavior.</p>

<p><strong>Edit:</strong> regarding whether the loss should be lower for the higher accuracy, it is going to depend on a couple of things. First, I am assuming that you are using softmax cross-entropy for your loss (as stated in the deep_cnn tutorial you linked), and assuming accuracy is the total number of correct predictions divided by the total number of samples. In this case, a lower loss <em>on the same dataset</em> should correlate to a higher accuracy. The emphasis is important.</p>

<p>If you are reporting loss during training but then report accuracy on your validation (or testing) dataset, it is possible for these two to be only loosely correlated. This is because the model is fitting (minimizing loss) to a certain subset of your total samples throughout the training process, and then tests against new samples that it has never seen before to verify that it generalizes well. The loss against this testing/validation set could be (and probably is) higher than the loss against the training set, so if the two numbers are being reported from different sets, you may not be able to draw comparisons like "loss for 1 GPU case should be lower since its accuracy is lower".</p>

<p>Second, if you are distributing the training then you are calculating losses across multiple workers (I believe), but only one accuracy at the end, again against a testing or validation set. Maybe the loss being reported is the best loss seen by any one worker, but overall the average losses were higher.</p>

<p>Basically I do not think we have enough information to decisively say why the loss and accuracy do not seem to correlate the way you expect, but there are a number of ways this could be happening, so I wouldn't dismiss it out of hand.</p>
Multi GPU architecture, gradient averaging - less accurate model?
<p>When I execute the cifar10 model as described at <a href="https://www.tensorflow.org/tutorials/deep_cnn" rel="noreferrer">https://www.tensorflow.org/tutorials/deep_cnn</a> I achieve 86% accuracy after approx 4 hours using a single GPU , when I utilize 2 GPU's the accuracy drops to 84% but reaching 84% accuracy is faster on 2 GPU's than 1. </p>

<p>My intuition is 
that average_gradients function as defined at <a href="https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py" rel="noreferrer">https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py</a> returns a less accurate gradient value as an average of gradients will be less accurate than the actual gradient value. </p>

<p>If the gradients are less accurate then the parameters than control the function that is learned as part of training is less accurate. Looking at the code (<a href="https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py" rel="noreferrer">https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py</a>) why is averaging the gradients over multiple GPU's less accurate than computing the gradient on a single GPU ?</p>

<p>Is my intuition of averaging the gradients producing a less accurate value correct ?</p>

<p>Randomness in the model is described as : </p>

<pre><code>The images are processed as follows:
They are cropped to 24 x 24 pixels, centrally for evaluation or randomly for training.
They are approximately whitened to make the model insensitive to dynamic range.
For training, we additionally apply a series of random distortions to artificially increase the data set size:

Randomly flip the image from left to right.
Randomly distort the image brightness.
Randomly distort the image contrast.
</code></pre>

<p>src : <a href="https://www.tensorflow.org/tutorials/deep_cnn" rel="noreferrer">https://www.tensorflow.org/tutorials/deep_cnn</a></p>

<p>Does this have an effect on training accuracy ?</p>

<p>Update : </p>

<p>Attempting to investigate this further, the loss function value training with different number of GPU's.</p>

<pre><code>Training with 1 GPU : loss value : .7 , Accuracy : 86%
Training with 2 GPU's : loss value : .5 , Accuracy : 84%
</code></pre>

<p>Shouldn't the loss value be lower for higher for higher accuracy, not vice versa ?</p>