title: Why freeze position embedding when distill roBERTa?
body: <p>I'm interested in huggingface's distillBERT work, by going through their code (<a href="https://github.com/huggingface/transformers/blob/master/examples/distillation/train.py" rel="nofollow noreferrer">https://github.com/huggingface/transformers/blob/master/examples/distillation/train.py</a>), I found that if use roBERTa as student model, they will freeze the position embedding, I'm wondering what is this for? </p>
  
  <pre><code>def freeze_pos_embeddings(student, args):
      if args.student_type == "roberta":
          student.roberta.embeddings.position_embeddings.weight.requires_grad = False
      elif args.student_type == "gpt2":
          student.transformer.wpe.weight.requires_grad = False
  </code></pre>
  
  <p>I understand the reason of freezing token_type_embeddings, because roBERTa never used segment embeddings, but why position embeddings?<br>
  Thanks a lot for the help!</p>