title: How to understand the bias term in language model head (when we tie the word embeddings)?
body: <p>I was learning the masked language modeling codebase in Huggingface Transformers. Just a question to understand the language model head.</p>
  <p>Here at the final linear layer where we project hidden size to vocab size (<a href="https://github.com/huggingface/transformers/blob/f2fbe4475386bfcfb3b83d0a3223ba216a3c3a91/src/transformers/models/bert/modeling_bert.py#L685-L702" rel="nofollow noreferrer">https://github.com/huggingface/transformers/blob/f2fbe4475386bfcfb3b83d0a3223ba216a3c3a91/src/transformers/models/bert/modeling_bert.py#L685-L702</a>).</p>
  <pre class="lang-py prettyprint-override"><code>self.decoder = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
  self.bias = nn.Parameter(torch.zeros(config.vocab_size))
  self.decoder.bias = self.bias
  </code></pre>
  <p>We set the <code>bias</code> term to zero at the moment. And later when we initialize the weight, we tie the weight of the linear layer and the word embedding.</p>
  <p>But we don't do such a thing for the bias term. I wonder how we can understand that and why we want to initialize the bias term as a zero vector.</p>
  <p><a href="https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L1060-L1079" rel="nofollow noreferrer">https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L1060-L1079</a></p>
  <pre class="lang-py prettyprint-override"><code>    def tie_weights(self):
          &quot;&quot;&quot;
          Tie the weights between the input embeddings and the output embeddings.
          If the `torchscript` flag is set in the configuration, can't handle parameter sharing so we are cloning the
          weights instead.
          &quot;&quot;&quot;
          if getattr(self.config, &quot;tie_word_embeddings&quot;, True):
              output_embeddings = self.get_output_embeddings()
              if output_embeddings is not None:
                  self._tie_or_clone_weights(output_embeddings, self.get_input_embeddings())
  
          if getattr(self.config, &quot;is_encoder_decoder&quot;, False) and getattr(self.config, &quot;tie_encoder_decoder&quot;, False):
              if hasattr(self, self.base_model_prefix):
                  self = getattr(self, self.base_model_prefix)
              self._tie_encoder_decoder_weights(self.encoder, self.decoder, self.base_model_prefix)
  
          for module in self.modules():
              if hasattr(module, &quot;_tie_weights&quot;):
                  module._tie_weights()
  </code></pre>
  <p>My understanding:</p>
  <ol>
  <li>Because the final linear weight accepts the hidden representations that have been transformed by several feed-forward layers. We might not be able to match them exactly, we need the bias term to somehow regularize them.</li>
  </ol>
  <p>As I'm not sure my understanding is accurate, I would like to seek your opinions.</p>