title: Using pre-trained sentence embeddings in PyTorch
body: <p><strong>[Original post, see below for edited]</strong></p>
  
  <p>I am new to PyTorch, and trying to perform a sentence classification task in it. </p>
  
  <p>I have averaged out the word embeddings in each sentence (glove embeddings) to form the sentence embedding. Hence, the dimension of each sentence embedding is the same. As per my understanding, since I already have the embeddings I don't need the embedding layer before using the LSTM. 
  My model is as follows:</p>
  
  <pre><code>import torch
  import torch.autograd as autograd
  import torch.nn as nn
  import torch.nn.functional as F
  import torch.optim as optim
  
  class LSTM1(nn.Module):
  
  def __init__(self,args):
      super(LSTM1,self).__init__()
      self.args = args
      self.hidden_dim = args.hidden_dim
      self.lstm = nn.LSTM(args.embed_dim, args.hidden_dim)
      self.hidden2tag = nn.Linear(args.hidden_dim, 2)
      self.hidden = self.init_hidden()
  
  def init_hidden(self):
      return (autograd.Variable(torch.zeros(1,1, self.hidden_dim).cuda()),
          autograd.Variable(torch.zeros(1,1, self.hidden_dim).cuda()))
  
  def forward(self,embeds):
      embeds = autograd.Variable(torch.from_numpy(embeds[0]).float().cuda())
      lstm_output, self.hidden = self.lstm(embeds.view(1, 1, -1), self.hidden)
      tag_space = self.hidden2tag(lstm_output.view(1, -1))
      scores = F.log_softmax(tag_space)
      return scores
  </code></pre>
  
  <p>And I pass the sentences in the form of embeddings as follows:</p>
  
  <pre><code>model = model.LSTM1(args).cuda()
  criterion = nn.NLLLoss()
  optimizer = optim.SGD(model.parameters(), lr=1e-5, momentum=0.9, weight_decay=1e-5)
  
  optimizer.zero_grad()
  
  for epoch in range(20):
      for i in range(len(sentences)):
          optimizer.zero_grad()
          model.hidden = model.init_hidden()
          target = prepare_targets(tag_phrase[i],tag_to_ix,1) #Gets a Variable(long tensor) for the target, single value (either 0 or 1)
          score = model(sentences[i]) #sentences[i] is the embedding of sentence i
  
          loss = criterion(score,target)
          loss.backward()
          optimizer.step()
  </code></pre>
  
  <p>My doubts:</p>
  
  <ol>
  <li>So the embedding goes into the model, where in the forward function it gets converted to a Variablle(float tensor), and hence an appropriate input for the LSTM. This is my understanding of things. Is this correct?</li>
  <li>I am backpropagating after every sentence, so does that make each sentence a separate batch? How do I split the sentences into batches and what changes do I need to make in the model then?</li>
  <li>What is the most appropriate way to use pre-trained sentence embeddings in PyTorch?</li>
  <li>If this code seems correct, then the problem I'm facing is that everything is getting classified into a single class out of the two. Any suggestions to resolve this error?</li>
  </ol>
  
  <p>Thank you.</p>
  
  <p><strong>[EDIT] Improved Code</strong></p>
  
  <pre><code>embeddings = torch.from_numpy(embeddings).float().cuda()
  args.embed_num=embeddings.size(0)
  args.embed_dim=embeddings.size(1)
  
  seq = [i for i in range(13000)]
  seq_tensor = torch.LongTensor(seq).cuda() #Index tensor corresponding to the embedding.
  
  target = prepare_targets(tag_phrase[:13000],tag_to_ix,1)
  
  train_data = torch.utils.data.TensorDataset(seq_tensor.cuda(),target.data.cuda())
  trainloader = torch.utils.data.DataLoader(train_data, batch_size=100, shuffle=True)
  
  model = model2.LSTM1(args,embeddings, 3).cuda()
  criterion = nn.NLLLoss()
  optimizer = optim.SGD(model.parameters(), lr=1e-4, momentum=0.9, weight_decay=1e-5)
  
  for epoch in range(1000):
      avg_loss=0.
      for i, data in enumerate(trainloader,0):
          seq, target = data
          seq_var, target = autograd.Variable(seq.cuda()), autograd.Variable(target.cuda())
          correct=0
          optimizer.zero_grad()
          model.hidden=model.init_hidden()
          score=model(seq_var)
          loss = criterion(score,target)
          loss.backward()
          optimizer.step()
          epoch_lis.append(epoch)
          losses.append(loss.data[0])
          _,predicted = torch.max(score.data,1)
          correct += (predicted == target.data).sum()
          print i, correct
  </code></pre>
  
  <p><strong>model:</strong> </p>
  
  <pre><code>class LSTM1(nn.Module):
  
      def __init__(self,args,embeddings, layers):
          super(LSTM1,self).__init__()
          self.num_layers=layers
          self.args = args
          self.hidden_dim = args.hidden_dim
          self.embed = nn.Embedding(args.embed_num, args.embed_dim)
          self.embed.weight = nn.Parameter(embeddings)
          self.lstm = nn.LSTM(args.embed_dim, args.hidden_dim, num_layers=self.num_layers)
          self.hidden2tag = nn.Linear(args.hidden_dim, 2)
          self.hidden = self.init_hidden()
  
      def init_hidden(self):
          return (autograd.Variable(torch.zeros(self.num_layers,1, self.hidden_dim).cuda()),
              autograd.Variable(torch.zeros(self.num_layers,1, self.hidden_dim).cuda()))
  
      def forward(self,sentence):
          embeds = self.embed(sentence)
          lstm_output, self.hidden = self.lstm(embeds.view(len(sentence), 1, -1), self.hidden)
          tag_space = self.hidden2tag(lstm_output.view(len(sentence), -1))
          scores = F.log_softmax(tag_space)
          return scores
  </code></pre>