title: Taking the last state from BiLSTM (BiGRU) in PyTorch
body: <p>After reading several articles, I am still quite confused about correctness of my implementation of getting last hidden states from BiLSTM.</p>
  
  <ol>
  <li><a href="https://towardsdatascience.com/understanding-bidirectional-rnn-in-pytorch-5bd25a5dd66" rel="noreferrer">Understanding Bidirectional RNN in PyTorch (TowardsDataScience)</a></li>
  <li><a href="https://discuss.pytorch.org/t/packedsequence-for-seq2seq-model/3907/2" rel="noreferrer">PackedSequence for seq2seq model (PyTorch forums)</a></li>
  <li><a href="https://stackoverflow.com/questions/48302810/whats-the-difference-between-hidden-and-output-in-pytorch-lstm">What's the difference between “hidden” and “output” in PyTorch LSTM? (StackOverflow)</a></li>
  <li><a href="https://discuss.pytorch.org/t/select-tensor-in-a-batch-of-sequences/8613" rel="noreferrer"> Select tensor in a batch of sequences (Pytorch formums)
  </a></li>
  </ol>
  
  <p>The approach from the last source (4) seems to be the cleanest for me, but I am still uncertain if I understood the thread correctly. Am I using the right final hidden states from LSTM and reversed LSTM? This is my implementation
  </p>
  
  <pre><code># pos contains indices of words in embedding matrix
  # seqlengths contains info about sequence lengths
  # so for instance, if batch_size is 2 and pos=[4,6,9,3,1] and 
  # seqlengths contains [3,2], we have batch with samples
  # of variable length [4,6,9] and [3,1]
  
  all_in_embs = self.in_embeddings(pos)
  in_emb_seqs = pack_sequence(torch.split(all_in_embs, seqlengths, dim=0))
  output,lasthidden = self.rnn(in_emb_seqs)
  if not self.data_processor.use_gru:
      lasthidden = lasthidden[0]
  # u_emb_batch has shape batch_size x embedding_dimension
  # sum last state from forward and backward  direction
  u_emb_batch = lasthidden[-1,:,:] + lasthidden[-2,:,:]
  </code></pre>
  
  <p>Is it correct?</p>