title: How to get a weights of Query, Key and Value projection matrices for each distinct head of BERT in HuggingFace implementation?
body: <p>I'm trying to get weights of Query, Key and Value projection matrices of each BERT's attention head. I thought that I can make this via using <code>model.state_dict()</code>, but it doesn't give me distinct parameters for each attention head. For example, for a small version of BertForSequenceClassifictation model with two layers and two heads on each, this dict has only the following keys:</p>
  <pre><code>bert.embeddings.position_ids
  bert.embeddings.word_embeddings.weight
  bert.embeddings.position_embeddings.weight
  bert.embeddings.token_type_embeddings.weight
  bert.embeddings.LayerNorm.weight
  bert.embeddings.LayerNorm.bias
  bert.encoder.layer.0.attention.self.query.weight
  bert.encoder.layer.0.attention.self.query.bias
  bert.encoder.layer.0.attention.self.key.weight
  bert.encoder.layer.0.attention.self.key.bias
  bert.encoder.layer.0.attention.self.value.weight
  bert.encoder.layer.0.attention.self.value.bias
  bert.encoder.layer.0.attention.output.dense.weight
  bert.encoder.layer.0.attention.output.dense.bias
  bert.encoder.layer.0.attention.output.LayerNorm.weight
  bert.encoder.layer.0.attention.output.LayerNorm.bias
  bert.encoder.layer.0.intermediate.dense.weight
  bert.encoder.layer.0.intermediate.dense.bias
  bert.encoder.layer.0.output.dense.weight
  bert.encoder.layer.0.output.dense.bias
  bert.encoder.layer.0.output.LayerNorm.weight
  bert.encoder.layer.0.output.LayerNorm.bias
  bert.encoder.layer.1.attention.self.query.weight
  bert.encoder.layer.1.attention.self.query.bias
  bert.encoder.layer.1.attention.self.key.weight
  bert.encoder.layer.1.attention.self.key.bias
  bert.encoder.layer.1.attention.self.value.weight
  bert.encoder.layer.1.attention.self.value.bias
  bert.encoder.layer.1.attention.output.dense.weight
  bert.encoder.layer.1.attention.output.dense.bias
  bert.encoder.layer.1.attention.output.LayerNorm.weight
  bert.encoder.layer.1.attention.output.LayerNorm.bias
  bert.encoder.layer.1.intermediate.dense.weight
  bert.encoder.layer.1.intermediate.dense.bias
  bert.encoder.layer.1.output.dense.weight
  bert.encoder.layer.1.output.dense.bias
  bert.encoder.layer.1.output.LayerNorm.weight
  bert.encoder.layer.1.output.LayerNorm.bias
  bert.pooler.dense.weight
  bert.pooler.dense.bias
  classifier.weight
  classifier.bias
  </code></pre>
  <p>I.e. it provides, for example, only one query weight matrix for zero layer ( <code>bert.encoder.layer.0.attention.self.query.weight</code> ). But zero layer has two heads, so to which of them does this matrix belong? And how to access the weights of the query projection matrix of the <strong>first</strong> and the <strong>second</strong> attention head of zero layer <strong>distinctively</strong>?</p>
  <p>P.S. I also tried to look at <code>model.parameters()</code>, but it seemingly gives the same matrices, just without names.</p>
  <p>P.P.S. I know how to access attention matrix for each head after applying the model to the data sample, but it is not what I need.</p>