title: Scalability issues related to distributed tensorflow
body: <p>I was running some experiments to test the scalability of distributed tensorflow. My goal is to check how is the speedup of using 4gpus on 2nodes (in total 8gpus) compared with using single gpu.</p>
  
  <p>I chose ResNet as the network structure and modified the ResNet example code from the official model zoos (github.com/tensorflow/models/tree/master/resnet) to be the one can do distributed training.
  In the experiments, ResNet 56 layers is used for training on CIFAR10 data set and the settings follow the original paper.</p>
  
  <p>First thing I would like to ensure is by using Synchronous SGD (SyncReplicasOptimizer), 8 gpus can get a similar curve as 1gpu in terms of <strong>epoch-to-accuracy</strong> figure. I used the same effective batch size (i.e., <code>batch size = 1gpu's batch size/the-number-of-worker</code>) on the two experiments.</p>
  
  <p>Please check the experimental figures</p>
  
  <p><a href="https://i.stack.imgur.com/pnNTc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pnNTc.png" alt="enter image description here"></a></p>
  
  <blockquote>
    <p>tf-0: single gpu version</p>
    
    <p>tf-1: distributed version (2ps, 8workers)</p>
    
    <p>x-axis: #epoch (per one means going through the whole training data once)</p>
    
    <p>y-axis: accuracy of CIFAR-10 <strong>testing data</strong></p>
  </blockquote>
  
  <p>The single gpu (<strong>tf-0</strong>) result is similar to the original paper and the ones reproduced by many other people (e.g., <a href="http://torch.ch/blog/2016/02/04/resnets.html" rel="nofollow noreferrer">http://torch.ch/blog/2016/02/04/resnets.html</a>, <a href="https://github.com/ppwwyyxx/tensorpack/tree/master/examples/ResNet" rel="nofollow noreferrer">https://github.com/ppwwyyxx/tensorpack/tree/master/examples/ResNet</a>, or more examples here <a href="https://github.com/KaimingHe/deep-residual-networks#third-party-re-implementations" rel="nofollow noreferrer">https://github.com/KaimingHe/deep-residual-networks#third-party-re-implementations</a>).</p>
  
  <p>However, you can observe that <strong>tf-1</strong> (8gpus curve) has a very different pattern, which is not what I expected. My understanding of Synchronous SGD is that optimizer accumulate <code>#workers</code> gradients and then apply to the variables. So the procedure should be exactly the same as single gpu if the same effective batch size is set.</p>
  
  <p>Interestingly, when I set #worker = 2 in the distributed version, the curve (<strong>tf-2</strong>) become close to the single gpu version. But 4gpus (<strong>tf-3</strong>) got closer to <strong>tf-1</strong>.</p>
  
  <p><a href="https://i.stack.imgur.com/D5Vup.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/D5Vup.png" alt="enter image description here"></a></p>
  
  <blockquote>
    <p>tf-2: distributed version (2ps, 2workers)</p>
    
    <p>tf-3: distributed version (1ps, 4workers)</p>
  </blockquote>
  
  <p>Here are some detail settings (basically follows the original paper):</p>
  
  <blockquote>
    <p>batch size=128, num_residual_units=9, relu_leakiness=0</p>
    
    <p>learning rate scheduler: 0.1 for 1-80 epoch, 0.01 for 81-120 epoch, 0.001 for 120-160 epoch </p>
  </blockquote>
  
  <p>Environments:</p>
  
  <blockquote>
    <p>Tensorflow version: 0.12.1</p>
    
    <p>Installed version of CUDA and cuDNN: CUDA 7.5, cuDNN 5.1</p>
  </blockquote>
  
  <p>The single gpu version code can be found <a href="http://github.com/tensorflow/models/blob/22036b6f44efeb648fe2b2b4551f2b982edf0f5f/resnet/resnet_main.py" rel="nofollow noreferrer">here</a> </p>
  
  <p>The code for distributed version can be found <a href="http://gist.github.com/infwinston/d01721d9609eef4a38b8e5e57719b194" rel="nofollow noreferrer">here</a> .</p>
  
  <p>Does anyone has ideas why it got gradually worse when #workers increase? Thanks.</p>
  
  <hr>
  
  <p><strong>Updated figure</strong></p>
  
  <p>According to commit ecdee38a (I think v0.12.1 includes this commit), tensorflow seems to use <strong>averaging</strong> instead of <strong>summing up</strong> gradients calculated by each worker. so I tried the one with 4x learning rate in 4gpus case. But got a even worse result.</p>
  
  <p><img src="https://i.stack.imgur.com/Ogwtp.png" alt="experiments of modified learning rate"></p>
  
  <blockquote>
    <p>1xlr: 1x learning rate (0.1/0.01/0.001)</p>
    
    <p>4xlr: 4x learning rate (0.4/0.04/0.004)</p>
  </blockquote>