<p>There is no reason not to fine-tune the GloVe embeddings in order to have better score for your final task, except if you have to keep a link with another model which uses the original embeddings (for interpreting your results for instance).</p>

<p>When fine-tuning the embeddings for your objective function, the word embeddings will (potentially) loose their initial properties (performing well for word similarity and analogy tasks).</p>

<p>Using word embeddings is just a way not to initialize with random vectors, so would it make sense to keep the random vectors fixed?</p>

<p>There are several articles which fine tune the word embeddings, for instance this one: <a href="https://arxiv.org/abs/1505.07931" rel="nofollow noreferrer">https://arxiv.org/abs/1505.07931</a></p>

<p>I made the assumption that you have enough training data. Otherwise it would be better to let the word embeddings fixed since it involves less parameters to train and thus avoids overfitting.</p>
Using torch.nn.Embedding for GloVe: should we fine-tune the embeddings or just use them as they are?
<p>while transfer learning / fine-tuning recent language models, such as BERT and XLNET, is by far a very common practice, how is this for GloVe? </p>

<p>Basically, I see two options when using GloVe to get dense vector representations that can be used by downstream NNs.</p>

<p>1) Fine-tune GloVe embeddings (in pytorch terms, gradient enabled)</p>

<p>2) Just use the embeddings without gradient.</p>

<p>For instance, given GloVe's embeddings matrix, I do</p>

<pre><code>embed = nn.Embedding.from_pretrained(torch.tensor(embedding_matrix, dtype=torch.float))
...
dense = nn.Linear(...)
</code></pre>

<p>Is it best practice to solely use GloVe to get vector representation (and only train the dense layer and potentially other layers) or would one fine-tune the embeddings matrix, too? </p>