title: Pytorch deep convolutional network does not converge on CIFAR10
body: <p>I copied the CIFAR10 sample network from PyTorch tutorial and added more layers, including BN. Even after 45 epochs, the network keeps achieving 68% classification accuracy on the test set. </p>
  
  <p>The network consists of:</p>
  
  <ul>
  <li>2 convolutional layers with 3x3 kernels (input size reduces from 32px to 28px)</li>
  <li>one max pooling layer (input size reduces from 28px to 14px)</li>
  <li>3 convolutional layers with 3x3 kernels (input size reduces from 14px to 8px)</li>
  <li>A fully connected network with 3 layers of 256->256->10 neurons</li>
  <li>batch normaliation is applied on all layers, including the convolutional layers, except for the last FC layer</li>
  <li>Relu is applied on all the convolutional layers and all the hidden FC layers</li>
  </ul>
  
  <p>Did I build/use anything improperly?</p>
  
  <pre><code>import torch
  import torchvision
  import torchvision.transforms as transforms
  import torch.nn as nn
  import torch.nn.functional as F
  import torch.optim as optim
  
  class Net(nn.Module):
      def __init__(self):
          super(Net, self).__init__()
          self.conv1_1 = nn.Conv2d(3, 16, 3)  # 32 -&gt; 30
          self.bn1_1 = nn.BatchNorm2d(16)
          self.conv1_2 = nn.Conv2d(16, 16, 3) # 30 - &gt; 28
          self.bn1_2 = nn.BatchNorm2d(16)
          self.pool = nn.MaxPool2d(2, 2)  # 28 -&gt; 14
          self.conv2_1 = nn.Conv2d(16, 16, 3) # 14 -&gt; 12
          self.bn2_1 = nn.BatchNorm2d(16)
          self.conv2_2 = nn.Conv2d(16, 16, 3) # 12 -&gt; 10
          self.bn2_2 = nn.BatchNorm2d(16)
          self.conv2_3 = nn.Conv2d(16, 16, 3) # 10 -&gt; 8
          self.bn2_3 = nn.BatchNorm2d(16)
          self.fc1 = nn.Linear(16 * 8 * 8, 256)
          self.bn4 = nn.BatchNorm1d(256)
          self.fc2 = nn.Linear(256, 256)
          self.bn5 = nn.BatchNorm1d(256)
          self.fc3 = nn.Linear(256, 10)
  
      def forward(self, x):
          x = F.relu(self.bn1_1(self.conv1_1(x)))
          x = self.pool(F.relu(self.bn1_2(self.conv1_2(x))))
          x = F.relu(self.bn2_1(self.conv2_1(x)))
          x = F.relu(self.bn2_2(self.conv2_2(x)))
          x = F.relu(self.bn2_3(self.conv2_3(x)))
          x = x.view(-1, 16 * 8 * 8)
          x = F.relu(self.bn4(self.fc1(x)))
          x = F.relu(self.bn5(self.fc2(x)))
          x = self.fc3(x)
          return x
  
  net = Net()
  device = 'cuda:0'
  net.to(device)
  criterion = nn.CrossEntropyLoss()
  optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
  
  transform = transforms.Compose(
          [transforms.ToTensor(),
           transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
  
  trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                              download=True, transform=transform)
  trainloader = torch.utils.data.DataLoader(trainset, batch_size=8,
                                                shuffle=True, num_workers=2)
  
  for epoch in range(128):  # loop over the dataset multiple times
      for i, data in enumerate(trainloader, 0):
          # get the inputs
          inputs, labels = data
          inputs, labels = inputs.to(device), labels.to(device)
  
          # zero the parameter gradients
          optimizer.zero_grad()
  
          # forward + backward + optimize
          outputs = net(inputs)
          loss = criterion(outputs, labels)
          loss.backward()
          optimizer.step()
  </code></pre>
  
  <p>Note: added the "Python" tag so the code gets highlighted</p>
  
  <p>Note: updated the <code>forward</code> method to apply <code>F.relu</code> on the hidden FC layers</p>