title: Pytorch using Multi GPU / accuracy is too low(10%)
body: <pre><code>if torch.cuda.is_available():
  for epoch in range(epoch_num):  
      for i,(images, labels) in enumerate(trainloader):
          images=images.to(device)
          labels=labels.to(device)
  
          optimizer.zero_grad()
  
          #Forward Backward Optimize
          outputs = net(images)
          loss = criterion(outputs, labels)
          loss.backward()
          optimizer.step()
  
          # print statistics
          if i%1000==0:
              print('Number of epochs: %d, Mini Batch order: %d' %(epoch+1,i))
  </code></pre>
  
  <p>This is a train part. I use four GPUs to train. And after I run 5000 epochs (batch size is 128), the accuracy comes out "10%"!! It's too low!
  Following is the test part:</p>
  
  <pre><code>with torch.no_grad():
  num_correct = 0
  total_data = 0
  if torch.cuda.is_available():
      for images, labels in testloader:
          images=images.to(device)
          labels=labels.to(device)
          output = net(images)
          _, expected = torch.max(output.data, 1)
  
          total_data += labels.size(0)
          num_correct += (expected == labels).sum().item()
  </code></pre>
  
  <p>I have no idea what is wrong, how can I investigate this?</p>