title: PyTorch Model is not training
body: <p>I have a problem that I can't fix already for a week. I am trying to build CIFAR-10 classifier, but my loss value after each batch is jumping randomly, and accuracy is not improving even on same batch (I can't even overfit model with one batch), so i guess the only possible reason is - weights are not updating. </p>
  
  <p><strong>My Module class</strong></p>
  
  <pre><code>class Net(nn.Module):
  def __init__(self):
      super(Net, self).__init__()
      self.conv_pool = nn.Sequential(
          nn.Conv2d(3, 64, 3, padding=1),
          nn.ReLU(),
          nn.MaxPool2d(2, 2),
          nn.Conv2d(64, 128, 3, padding=1),
          nn.ReLU(),
          nn.MaxPool2d(2, 2),
          nn.Conv2d(128, 256, 3, padding=1),
          nn.ReLU(),
          nn.MaxPool2d(2, 2),
          nn.Conv2d(256, 512, 3, padding=1),
          nn.ReLU(),
          nn.MaxPool2d(2, 2),
          nn.Conv2d(512, 512, 1),
          nn.ReLU(),
          nn.MaxPool2d(2, 2))
  
      self.fcnn = nn.Sequential(
          nn.Linear(512, 2048),
          nn.ReLU(),
          nn.Linear(2048, 2048),
          nn.ReLU(),
          nn.Linear(2048, 10)
      )
  
  def forward(self, x):
      x = self.conv_pool(x)
      x = x.view(-1, 512)
      x = self.fcnn(x)
      return x
  </code></pre>
  
  <p>Optimizers I am using: </p>
  
  <pre><code>net = Net()
  criterion = nn.CrossEntropyLoss()
  optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
  </code></pre>
  
  <p>My Train function: </p>
  
  <pre><code>def train():
  for epoch in range(5):  # loop over the dataset multiple times
      for i in range(0, df_size):
          # get the data
  
          try:
              images, labels = loadBatch(ds, i)
          except BaseException:
              continue
  
          # wrap 
          inputs = Variable(images)
  
          optimizer.zero_grad()
  
          outputs = net(inputs)
  
          loss = criterion(outputs, Variable(labels))
  
          loss.backward()
          optimizer.step()
          acc = test(images,labels)
          print("Loss: " + str(loss.data[0]) + " Accuracy %: " + str(acc) + " Iteration: " + str(i))
  
          if i % 40 == 39:
              torch.save(net.state_dict(), "model_save_cifar")
  
      print("Finished epoch " + str(epoch))
  </code></pre>
  
  <p>I am using <strong>batch_size</strong> = 20, <strong>image_size</strong> = 32 (CIFAR-10)</p>
  
  <p><strong>loadBatch</strong> functions returns tuple of <strong>LongTensor</strong> 20x3x32x32 for images and <strong>LongTensor</strong> 20x1 for labels</p>
  
  <p>I would be really happy if you can help me, or suggest possible solution ( I have a guess that it's because sequential modules in NN, but parameters that I am passing to optimizer seems to be right)</p>