title: Pytorch incorrect value of member variable when using Multi-gpu
body: <p>Here is a simple class for running in multi-gpu environment. The member variable <code>self.firstIter</code> should be <code>False</code> after the first iteration.</p>
  
  <pre class="lang-py prettyprint-override"><code>Class TestNetwork(nn.Module):
  
      def __init__(self):
          super(TestNetwork, self).__init__()
          self.firstIter = True #indicates whether it's the first iteration
  
      def forward(self, input):
          print 'is firstIter: ', self.firstIter #always True!!
          if self.firstIter is True:
              self.firstIter = False
          # do otherthings
  </code></pre>
  
  <p>The code works as expected when using only one gpu. </p>
  
  <p>However when using multi-gpu (i.e. <code>nn.DataParallel</code>), the value of <code>self.firstIter</code> is always printed as <code>True</code>. </p>
  
  <p><strong>Why does this happen? What is wrong with the code?</strong> </p>
  
  <p><em>Using PyTorch version 0.3.1.</em></p>