title: Why we need the init_weight function in BERT pretrained model in Huggingface Transformers?
body: <p>In the code by Hugginface transformers, there are many fine-tuning models have the function <code>init_weight</code>. 
  For example(<a href="https://github.com/huggingface/transformers/blob/a9aa7456ac/src/transformers/modeling_bert.py#L1073-L1082" rel="nofollow noreferrer">here</a>), there is a <code>init_weight</code> function at last.</p>
  
  <pre class="lang-py prettyprint-override"><code>class BertForSequenceClassification(BertPreTrainedModel):
      def __init__(self, config):
          super().__init__(config)
          self.num_labels = config.num_labels
  
          self.bert = BertModel(config)
          self.dropout = nn.Dropout(config.hidden_dropout_prob)
          self.classifier = nn.Linear(config.hidden_size, config.num_labels)
  
          self.init_weights()
  
  </code></pre>
  
  <p>As I know, it will call the following <a href="https://github.com/huggingface/transformers/blob/a9aa7456ac/src/transformers/modeling_bert.py#L520-L530" rel="nofollow noreferrer">code</a></p>
  
  <pre class="lang-py prettyprint-override"><code>def _init_weights(self, module):
      """ Initialize the weights """
      if isinstance(module, (nn.Linear, nn.Embedding)):
          # Slightly different from the TF version which uses truncated_normal for initialization
          # cf https://github.com/pytorch/pytorch/pull/5617
          module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
      elif isinstance(module, BertLayerNorm):
          module.bias.data.zero_()
          module.weight.data.fill_(1.0)
      if isinstance(module, nn.Linear) and module.bias is not None:
          module.bias.data.zero_()
  </code></pre>
  
  <p>My question is <strong>If we are loading the pre-trained model, why do we need to initialize the weight for every module?</strong></p>
  
  <p>I guess I must be misunderstanding something here.</p>