title: PyTorch naive single label classification with embedding layer fails at random
body: <p>I am new to PyTorch and I am trying out the Embedding Layer. </p>
  
  <p>I wrote a naive classification task, where all the inputs are the equal and all the labels are set to 1.0. I hence expect the model to learn quickly to predict 1.0.</p>
  
  <p>The input is always 0, which is fed into a nn.Embedding(1,32) layer, followed by nn.Linear(32,1) and nn.Relu().</p>
  
  <p>However, an unexpected and undesired behavior occurs: training outcome is different for different times I run the code.
  For example, </p>
  
  <ul>
  <li>setting the random seed to 10, model converges: loss decreases and model always predicts 1.0</li>
  <li>setting the random seed to 1111, model doesn't converge: loss doesn't decrease and model always predicts 0.5. In those cases the parameters are not updated</li>
  </ul>
  
  <p>Here is the minimal, replicable code:</p>
  
  <pre><code>from torch.nn import BCEWithLogitsLoss
  import torch.nn as nn
  import torch.optim as optim
  from torch.utils.data import DataLoader
  from torch.autograd import Variable
  from torch.utils.data import Dataset
  import torch
  
  
  class MyModel(nn.Module):
  
      def __init__(self):
          super(MyModel, self).__init__()
          self.vgg_fc = nn.Linear(32, 1)
          self.relu = nn.ReLU()
          self.embeddings = nn.Embedding(1, 32)
  
      def forward(self, data):
          emb = self.embeddings(data['index'])
          return self.relu(self.vgg_fc(emb))
  
  
  class MyDataset(Dataset):
  
      def __init__(self):
          pass
      def __len__(self):
          return 1000
      def __getitem__(self, idx):
          return {'label': 1.0, 'index': 0}
  
  
  def train():
      model = MyModel()
      db = MyDataset()
      dataloader = DataLoader(db, batch_size=256, shuffle=True, num_workers=16)
  
      loss_function = BCEWithLogitsLoss()
      optimizer_rel = optim.SGD(model.parameters(), lr=0.1)
  
      for epoch in range(50):
          for i_batch, sample_batched in enumerate(dataloader):
  
              model.zero_grad()
              out = model({'index': Variable(sample_batched['index'])})
  
              labels = Variable(sample_batched['label'].type(torch.FloatTensor).view(sample_batched['label'].shape[0], 1))
  
              loss = loss_function(out, labels)
              loss.backward()
              optimizer_rel.step()
              print 'Epoch:', epoch, 'batch', i_batch, 'Tr_Loss:', loss.data[0]
      return model
  
  
  if __name__ == '__main__':
  
      # please, try seed 10 (converge) and seed 1111 (fails)
      torch.manual_seed(10)
      train()
  </code></pre>
  
  <p>Without specifying the random seed, different runs have different outcome. </p>
  
  <p>Why is, in those cases, the model unable to learn such a easy task?
  Is there any mistake in the way I use nn.Embedding layer?</p>
  
  <p>Thank you </p>