<p>In most (maybe even all) commonly used Transformers, position embeddings are not trained but defined using an analytically described function (unnumbered equation on 
 page 6 of <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">Attention is all you need</a> paper):</p>

<p><a href="https://i.stack.imgur.com/OEYeo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OEYeo.png" alt="enter image description here"></a></p>

<p>To save the computation time in the <a href="https://github.com/huggingface/transformers" rel="nofollow noreferrer">Transformer package</a>, they are pre-computed up to the length of 512 and stored as a variable that serves as a cache which should not change during training.</p>

<p>The reason for not training the position embeddings is that the embeddings for later positions would be undertrained, but with cleverly analytically defined position embeddings, the network can learn the regularity behind the equations and generalize for longer sequences more easily.</p>
Why freeze position embedding when distill roBERTa?
<p>I'm interested in huggingface's distillBERT work, by going through their code (<a href="https://github.com/huggingface/transformers/blob/master/examples/distillation/train.py" rel="nofollow noreferrer">https://github.com/huggingface/transformers/blob/master/examples/distillation/train.py</a>), I found that if use roBERTa as student model, they will freeze the position embedding, I'm wondering what is this for? </p>

<pre><code>def freeze_pos_embeddings(student, args):
    if args.student_type == "roberta":
        student.roberta.embeddings.position_embeddings.weight.requires_grad = False
    elif args.student_type == "gpt2":
        student.transformer.wpe.weight.requires_grad = False
</code></pre>

<p>I understand the reason of freezing token_type_embeddings, because roBERTa never used segment embeddings, but why position embeddings?<br>
Thanks a lot for the help!</p>