<p>Let's say you would like to add <a href="https://arxiv.org/abs/1710.05941" rel="nofollow noreferrer"><code>swish</code></a> or <a href="https://arxiv.org/abs/1606.08415" rel="nofollow noreferrer"><code>gelu</code></a> to keras, the previous methods are nice inline insertions. But you could also insert them in the set of keras activation functions, so that you call you custom fucntion as you would call <code>ReLU</code>. I tested this with keras 2.2.2 (any v2 would do). Append to this file <code>$HOME/anaconda2/lib/python2.7/site-packages/keras/activations.py</code> the definition of your custom function (can be different for you python and anaconda version).</p>

<p>In keras internal:</p>

<pre><code>$HOME/anaconda2/lib/python2.7/site-packages/keras/activations.py

def swish(x):
    return (K.sigmoid(beta * x) * alpha *x)
</code></pre>

<p>Then in your python file:</p>

<pre><code>$HOME/Documents/neural_nets.py

model = Sequential()
model.add(Activation('swish'))
</code></pre>
How do you create a custom activation function with Keras?
<p>Sometimes the default <a href="https://keras.io/activations/" rel="noreferrer">standard activations</a> like ReLU, tanh, softmax, ... and the <a href="https://keras.io/layers/advanced-activations/" rel="noreferrer">advanced activations</a> like LeakyReLU aren't enough. And it might also not be in <a href="https://github.com/farizrahman4u/keras-contrib" rel="noreferrer">keras-contrib</a>.</p>

<p>How do you create your own activation function?</p>