<p>This is a very good question, and shows a common misconception about Transformers, stemming from an (unfortunate) formulation in the <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">original Transformers paper</a>. In particular, the authors write the following in Section 3.2.2:</p>
<blockquote>
<p>In this work, we employ <code>h = 8</code> parallel attention layers, or heads. For each of these <strong>we use</strong> <code>d_k = d_v = d_(model) / h = 64</code>. [...]</p>
</blockquote>
<p>Note that the equality of <code>d_k/d_v = d_(model)</code> is not strictly necessary; it is only important that you do match the final hidden representation (<code>d_(model)</code>) <em>after the Feed-Forward portion of each layer</em>. Specifically for <code>mt5-small</code>, the authors actually use an internal dimension of <code>384</code> which is simply the product of parameters <code>d_kv * num_heads = 64 * 6</code>.</p>
<p>Now, the problem is that many libraries make a similar assumption of the enforced relation between <code>d_kv</code> and <code>d_(model)</code>, because it saves some implementation effort that most people won't use anyways. I suspect (not super familiar with AllenNLP) that they have made similar assumptions here, which is why you cannot load the model.</p>
<p>Also, to clarify this, here is a peek at the <code>modules</code> of a loaded <code>mt5-small</code>:</p>
<pre class="lang-py prettyprint-override"><code>T5Block(
    (layer): ModuleList(
        (0): T5LayerSelfAttention(
        (SelfAttention): T5Attention(
            (q): Linear(in_features=512, out_features=384, bias=False)
            (k): Linear(in_features=512, out_features=384, bias=False)
            (v): Linear(in_features=512, out_features=384, bias=False)
            (o): Linear(in_features=384, out_features=512, bias=False)
        )
        (layer_norm): T5LayerNorm()
        (dropout): Dropout(p=0.1, inplace=False)
        )
        (1): T5LayerFF(
        (DenseReluDense): T5DenseGatedGeluDense(
            (wi_0): Linear(in_features=512, out_features=1024, bias=False)
            (wi_1): Linear(in_features=512, out_features=1024, bias=False)
            (wo): Linear(in_features=1024, out_features=512, bias=False)
            (dropout): Dropout(p=0.1, inplace=False)
        )
        (layer_norm): T5LayerNorm()
        (dropout): Dropout(p=0.1, inplace=False)
        )
    )
)

</code></pre>
<p>You can get the full model layout by simply calling <code>list(model.modules())</code></p>
Google mT5-small configuration error because number attention heads is not divider of model dimension
<p>The configuration file for the HuggingFace <strong>google/mt5-small</strong> Model (<a href="https://huggingface.co/google/mt5-small" rel="nofollow noreferrer">https://huggingface.co/google/mt5-small</a>)</p>
<p>defines</p>
<pre><code>{
...
  &quot;d_model&quot;: 512,
...
  &quot;num_heads&quot;: 6,
...
}
</code></pre>
<p>Link to the config file: <a href="https://huggingface.co/google/mt5-small/resolve/main/config.json" rel="nofollow noreferrer">https://huggingface.co/google/mt5-small/resolve/main/config.json</a></p>
<p><strong>Question:</strong></p>
<p>As far as I understood, the number of attention-head should be a divider of the model dimension. This is clearly not true in this config file.</p>
<p>Do I misunderstand how self-attention is applied in mT5?</p>
<p>When I use the AllenNLP model (<a href="https://github.com/allenai/allennlp-models/blob/main/allennlp_models/generation/models/t5.py" rel="nofollow noreferrer">https://github.com/allenai/allennlp-models/blob/main/allennlp_models/generation/models/t5.py</a>)
as sequence-to-sequence model, I receive an error message</p>
<p><strong>Summary:</strong></p>
<pre><code>allennlp.common.checks.ConfigurationError: The hidden size (512) is not a multiple of the number of attention heads (6)
</code></pre>
<p><strong>Full</strong></p>
<pre><code>Traceback (most recent call last):
  File &quot;/snap/pycharm-professional/269/plugins/python/helpers/pydev/pydevd.py&quot;, line 1500, in _exec
    runpy._run_module_as_main(module_name, alter_argv=False)
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/runpy.py&quot;, line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/runpy.py&quot;, line 87, in _run_code
    exec(code, run_globals)
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/__main__.py&quot;, line 50, in &lt;module&gt;
    run()
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/__main__.py&quot;, line 46, in run
    main(prog=&quot;allennlp&quot;)
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/commands/__init__.py&quot;, line 123, in main
    args.func(args)
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/commands/train.py&quot;, line 112, in train_model_from_args
    train_model_from_file(
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/commands/train.py&quot;, line 178, in train_model_from_file
    return train_model(
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/commands/train.py&quot;, line 254, in train_model
    model = _train_worker(
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/commands/train.py&quot;, line 490, in _train_worker
    train_loop = TrainModel.from_params(
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/common/from_params.py&quot;, line 652, in from_params
    return retyped_subclass.from_params(
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/common/from_params.py&quot;, line 686, in from_params
    return constructor_to_call(**kwargs)  # type: ignore
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/commands/train.py&quot;, line 766, in from_partial_objects
    model_ = model.construct(
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/common/lazy.py&quot;, line 82, in construct
    return self.constructor(**contructor_kwargs)
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/common/lazy.py&quot;, line 66, in constructor_to_use
    return self._constructor.from_params(  # type: ignore[union-attr]
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/common/from_params.py&quot;, line 652, in from_params
    return retyped_subclass.from_params(
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/common/from_params.py&quot;, line 686, in from_params
    return constructor_to_call(**kwargs)  # type: ignore
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp_models/generation/models/t5.py&quot;, line 32, in __init__
    self.t5 = T5Module.from_pretrained_module(
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/modules/transformer/transformer_module.py&quot;, line 251, in from_pretrained_module
    model = cls._from_config(config, **kwargs)
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/modules/transformer/t5.py&quot;, line 852, in _from_config
    return cls(
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/modules/transformer/t5.py&quot;, line 783, in __init__
    self.encoder: T5EncoderStack = encoder.construct(
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/common/lazy.py&quot;, line 82, in construct
    return self.constructor(**contructor_kwargs)
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/modules/transformer/t5.py&quot;, line 600, in basic_encoder
    self_attention=block_self_attention.construct(
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/common/lazy.py&quot;, line 82, in construct
    return self.constructor(**contructor_kwargs)
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/common/lazy.py&quot;, line 66, in constructor_to_use
    return self._constructor.from_params(  # type: ignore[union-attr]
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/common/from_params.py&quot;, line 686, in from_params
    return constructor_to_call(**kwargs)  # type: ignore
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/modules/transformer/attention_module.py&quot;, line 471, in __init__
    super().__init__(
  File &quot;/home/lars/anaconda3/envs/mare2/lib/python3.9/site-packages/allennlp/modules/transformer/attention_module.py&quot;, line 91, in __init__
    raise ConfigurationError(
allennlp.common.checks.ConfigurationError: The hidden size (512) is not a multiple of the number of attention heads (6)
</code></pre>