<p>The number of attention heads is irrespective of the number of (encoder) layers.
However, there is an inherent tie between the hidden size of each model (768 for <code>bert-base</code>, and 1024 for <code>bert-large</code>), which is explained in <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">the original Transformers paper</a>.
Essentially, the choice made by the authors is that the self-attention block size (<code>d_k</code>) equals the hidden dimension (<code>d_hidden</code>), divided by the number of heads (<code>h</code>), or formally</p>
<pre><code>d_k = d_hidden / h
</code></pre>
<p>Since the standard choice seems to be <code>d_k = 64</code>, we can infer the final size from our parameters:</p>
<pre><code>h = d_hidden / d_k = 1024 / 64 = 16
</code></pre>
<p>which is exactly the value you are looking at in <code>bert-large</code>.</p>
BERT Heads Count
<p>From the literature I read,</p>
<p>Bert Base has 12 encoder layers and 12 attention heads.  Bert Large has 24 encoder layers and 16 attention heads.</p>
<p>Why is Bert large having 16 attentions heads ?</p>