title: Using torch.nn.Embedding for GloVe: should we fine-tune the embeddings or just use them as they are?
body: <p>while transfer learning / fine-tuning recent language models, such as BERT and XLNET, is by far a very common practice, how is this for GloVe? </p>
  
  <p>Basically, I see two options when using GloVe to get dense vector representations that can be used by downstream NNs.</p>
  
  <p>1) Fine-tune GloVe embeddings (in pytorch terms, gradient enabled)</p>
  
  <p>2) Just use the embeddings without gradient.</p>
  
  <p>For instance, given GloVe's embeddings matrix, I do</p>
  
  <pre><code>embed = nn.Embedding.from_pretrained(torch.tensor(embedding_matrix, dtype=torch.float))
  ...
  dense = nn.Linear(...)
  </code></pre>
  
  <p>Is it best practice to solely use GloVe to get vector representation (and only train the dense layer and potentially other layers) or would one fine-tune the embeddings matrix, too? </p>