<p>Transformers were originally proposed, as the title of &quot;Attention is All You Need&quot; implies, as a more efficient seq2seq model ablating the RNN structure commonly used til that point.</p>
<p>However in pursuing this efficiency, a single headed attention had reduced descriptive power compared to RNN based models. Multiple heads were proposed to mitigate this, allowing the model to learn multiple lower-scale feature maps as opposed to one all-encompasing map:</p>
<blockquote>
<p>In these models, the number of operations required to relate signals from two arbitrary input or output positions grows in the distance between positions [...] This makes it more difficult to learn dependencies between distant positions [12]. In the Transformer this is reduced to a constant number of operations, albeit at the cost of reduced effective resolution due to averaging attention-weighted positions, an effect we counteract with Multi-Head Attention...</p>
<ul>
<li><a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention is All You Need</a> (2017)</li>
</ul>
</blockquote>
<p>As such, <a href="https://ai.stackexchange.com/a/26840/23503"><strong>multiple attention heads</strong></a> in a single layer in a transformer <strong>is analogous to multiple kernels in a single layer in a CNN</strong>: they have the same architecture, and operate on the same feature-space, but since they are separate 'copies' with different sets of weights, they are hence 'free' to learn different functions.</p>
<p>In a CNN this may correspond to different definitions of visual features, and in a Transformer this may correspond to different definitions of relevance:<sup>1</sup></p>
<p>For example:</p>
<div class="s-table-container">
<table class="s-table">
<thead>
<tr>
<th>Architecture</th>
<th>Input</th>
<th>(Layer 1)<br>Kernel/Head 1</th>
<th>(Layer 1)<br>Kernel/Head 2</th>
</tr>
</thead>
<tbody>
<tr>
<td>CNN</td>
<td>Image</td>
<td>Diagonal <a href="https://en.wikipedia.org/wiki/Kernel_(image_processing)#Details" rel="nofollow noreferrer">edge-detection</a></td>
<td>Horizontal edge-detection</td>
</tr>
<tr>
<td>Transformer</td>
<td>Sentence</td>
<td>Attends to next word</td>
<td>Attends from verbs to their direct objects</td>
</tr>
</tbody>
</table>
</div><hr />
<p><strong>Notes:</strong></p>
<sup>
<ol>
<li>There is no guarantee that these are human interpretable, but in many popular architectures they do map accurately onto linguistic concepts:
<blockquote>
<p>While no single head performs well at many relations, we find that particular heads correspond remarkably well to particular relations. For example, we find heads that find direct objects of verbs, determiners of nouns, objects of prepositions, and objects of possesive pronouns...</p>
<ul>
<li><a href="https://www.aclweb.org/anthology/W19-4828/" rel="nofollow noreferrer">What Does BERT Look at? An Analysis of BERT’s Attention</a> (2019)</li>
</ul>
</blockquote>
</li>
</ol>
</sup>
Why use multi-headed attention in Transformers?
<p>I am trying to understand why transformers use multiple attention heads. I found the following <a href="https://towardsdatascience.com/simple-explanation-of-transformers-in-nlp-da1adfc5d64f" rel="nofollow noreferrer">quote</a>:</p>
<blockquote>
<p>Instead of using a single attention function where the attention can
be dominated by the actual word itself, transformers use multiple
attention heads.</p>
</blockquote>
<p>What is meant by &quot;the attention being dominated by the word itself&quot; and how does the use of multiple heads address that?</p>