<p>GELU is a smoother version of the RELU.</p>
<p>ReLU vs GELU:</p>
<p><a href="https://i.stack.imgur.com/17HiP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/17HiP.png" alt="enter image description here" /></a></p>
<p>I think the reason is stated in <a href="https://arxiv.org/pdf/1606.08415.pdf#page=6" rel="nofollow noreferrer">the paper</a>:</p>
<p><a href="https://i.stack.imgur.com/LVPZN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LVPZN.png" alt="enter image description here" /></a></p>
Why "GELU" activation function is used insread of ReLu in BERT?
<p>I have in the activation function <strong>Gaussian Error Linear Units(GELUs)</strong> is used in the popular NLP model <strong><em>BERT</em></strong>. Is there any solid reason ?</p>