<p>The author of the <a href="https://arxiv.org/abs/1810.04805" rel="nofollow noreferrer">original BERT paper</a> answered it (kind of) in a <a href="https://github.com/google-research/bert/issues/43#issuecomment-435980269" rel="nofollow noreferrer">comment on GitHub</a>.</p>
<blockquote>
<p>The tanh() thing was done early to try to make it more interpretable but it probably doesn't matter either way.</p>
</blockquote>
<p>I agree it doesn't fully answer &quot;whether&quot; <code>tanh</code> is preferable, but from the looks of it, it'll probably work with any activation.</p>
why do pooler use tanh as a activation func in bert, rather than gelu?
<p>class BERTPooler(nn.Module):
def <strong>init</strong>(self, config):
super(BERTPooler, self).<strong>init</strong>()
self.dense = nn.Linear(config.hidden_size, config.hidden_size)
self.activation = nn.Tanh()</p>
<pre><code>def forward(self, hidden_states):
    # We &quot;pool&quot; the model by simply taking the hidden state corresponding
    # to the first token.
    first_token_tensor = hidden_states[:, 0]
    pooled_output = self.dense(first_token_tensor)
    pooled_output = self.activation(pooled_output)
    return pooled_output
</code></pre>